Nov 03 08:36:48.504448 ip-10-0-131-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 08:36:48.504459 ip-10-0-131-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 08:36:48.504576 ip-10-0-131-117 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 08:36:48.504797 ip-10-0-131-117 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 08:36:58.622893 ip-10-0-131-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 08:36:58.622909 ip-10-0-131-117 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c7773d95afce483e935fbbf241c6f5bc -- Nov 03 08:38:54.740462 ip-10-0-131-117 systemd[1]: Starting Kubernetes Kubelet... Nov 03 08:38:55.125310 ip-10-0-131-117 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:55.125310 ip-10-0-131-117 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 08:38:55.125310 ip-10-0-131-117 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:55.125310 ip-10-0-131-117 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:38:55.128937 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128833 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.128937 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128936 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.128937 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128941 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.128937 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128944 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128948 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128951 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128954 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128956 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128959 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128962 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128964 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128967 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128969 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128971 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128974 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128977 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128979 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128982 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.128985 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129007 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129010 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129013 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129016 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.129068 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129019 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129021 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129024 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129026 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129029 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129032 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129034 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129037 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129039 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129042 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129044 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129047 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129052 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129057 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129061 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129063 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129066 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129069 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129072 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.129606 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129074 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129077 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129080 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129084 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129087 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129089 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129092 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129094 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129097 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129099 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129103 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129107 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129111 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129113 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129116 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129118 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129121 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129123 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129127 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.130100 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129129 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129132 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129149 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129153 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129157 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129162 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129166 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129170 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129175 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129179 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129182 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129185 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129188 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129190 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129193 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129195 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129198 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129201 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129204 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.130759 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129207 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129209 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129212 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129215 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129217 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129220 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129222 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129225 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129227 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129230 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129232 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129235 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129238 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129240 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129243 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129246 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129249 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129251 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129254 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129258 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129261 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.131372 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129263 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129266 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129268 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129271 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.129274 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131555 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131575 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131579 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131583 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131585 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131589 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131592 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131596 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131599 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131601 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131604 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131607 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131614 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131617 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.131857 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131619 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131622 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131625 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131627 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131630 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131632 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131635 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131637 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131640 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131642 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131645 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131654 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131657 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131660 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131663 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131665 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131667 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131671 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131673 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131677 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.132334 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131680 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131682 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131685 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131688 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131690 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131693 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131695 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131698 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131701 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131703 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131705 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131708 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131710 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131713 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131715 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131719 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131723 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131725 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131728 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.132809 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131731 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131733 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131736 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131738 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131741 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131744 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131746 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131749 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131751 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131754 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131759 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131762 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131766 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131769 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131771 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131774 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131777 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131780 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131782 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131785 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.133308 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131787 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131790 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131792 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131795 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131797 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131800 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131802 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131805 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131809 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131812 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131816 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131819 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131822 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131824 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131827 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131829 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131832 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131834 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131837 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.133808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131839 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131842 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131844 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131848 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131851 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131854 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131857 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131860 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131863 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131866 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131868 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131871 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131873 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.131876 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133172 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133181 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133184 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133188 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133191 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133194 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.134303 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133196 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133199 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133202 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133207 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133210 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133213 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133216 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133219 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133221 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133224 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133227 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133229 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133232 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133234 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133237 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133240 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133243 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133246 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133248 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133251 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133254 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.134773 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133256 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133258 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133261 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133263 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133266 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133268 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133271 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133273 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133275 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133278 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133280 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133282 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133285 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133287 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133289 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133292 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133295 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133298 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133300 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.135295 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133302 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133305 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133307 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133310 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133312 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133315 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133318 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133321 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133324 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133327 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133329 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133332 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133334 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133338 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133342 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133345 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133348 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133351 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133353 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.135755 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133355 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133358 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133362 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133364 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133367 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133369 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133371 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133374 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133376 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133378 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133381 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133384 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133386 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133389 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133391 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133393 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133396 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133400 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133403 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133406 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.136211 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133409 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133411 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133414 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133417 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133420 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133422 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133424 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133427 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133430 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133432 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133435 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133437 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133440 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133442 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133444 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133447 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133449 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133452 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133454 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.136684 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133457 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.133461 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133560 2562 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133569 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133581 2562 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133586 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133591 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133594 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133599 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133604 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133607 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133610 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133613 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133617 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133620 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133623 2562 flags.go:64] FLAG: --cgroup-root="" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133626 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133629 2562 flags.go:64] FLAG: --client-ca-file="" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133632 2562 flags.go:64] FLAG: --cloud-provider="external" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133635 2562 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133640 2562 flags.go:64] FLAG: --cluster-domain="" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133643 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133646 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133650 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 08:38:55.137131 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133653 2562 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133661 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133665 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133668 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133671 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133674 2562 flags.go:64] FLAG: --contention-profiling="false" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133677 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133680 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133683 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133686 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133690 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133693 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133696 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133700 2562 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133704 2562 flags.go:64] FLAG: --enable-server="true" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133706 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133714 2562 flags.go:64] FLAG: --event-burst="100" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133718 2562 flags.go:64] FLAG: --event-qps="50" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133721 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133725 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133728 2562 flags.go:64] FLAG: --eviction-hard="" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133732 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133735 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133740 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133743 2562 flags.go:64] FLAG: --eviction-soft="" Nov 03 08:38:55.137705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133746 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133749 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133752 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133755 2562 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133758 2562 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133761 2562 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133763 2562 flags.go:64] FLAG: --feature-gates="" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133768 2562 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133771 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133774 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133777 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133780 2562 flags.go:64] FLAG: --healthz-port="10248" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133783 2562 flags.go:64] FLAG: --help="false" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133786 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133790 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133793 2562 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133796 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133799 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133802 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133805 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133808 2562 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133812 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133815 2562 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 08:38:55.138408 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133820 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133823 2562 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133826 2562 flags.go:64] FLAG: --kube-reserved="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133828 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133831 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133834 2562 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133837 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133841 2562 flags.go:64] FLAG: --lock-file="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133844 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133847 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133850 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133856 2562 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133859 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133862 2562 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133865 2562 flags.go:64] FLAG: --logging-format="text" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133869 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133872 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133875 2562 flags.go:64] FLAG: --manifest-url="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133878 2562 flags.go:64] FLAG: --manifest-url-header="" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133883 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133886 2562 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133890 2562 flags.go:64] FLAG: --max-pods="110" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133893 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133896 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133899 2562 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 08:38:55.138964 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133902 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133905 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133908 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133911 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133918 2562 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133921 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133924 2562 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133929 2562 flags.go:64] FLAG: --pod-cidr="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133932 2562 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133935 2562 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133938 2562 flags.go:64] FLAG: --pods-per-core="0" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133941 2562 flags.go:64] FLAG: --port="10250" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133944 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133947 2562 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a2369d01d0fa5665" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133951 2562 flags.go:64] FLAG: --qos-reserved="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133954 2562 flags.go:64] FLAG: --read-only-port="10255" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133957 2562 flags.go:64] FLAG: --register-node="true" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133960 2562 flags.go:64] FLAG: --register-with-taints="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133964 2562 flags.go:64] FLAG: --registry-burst="10" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133967 2562 flags.go:64] FLAG: --registry-qps="5" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133970 2562 flags.go:64] FLAG: --reserved-cpus="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133973 2562 flags.go:64] FLAG: --reserved-memory="" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133978 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133981 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133984 2562 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133987 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 08:38:55.139600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133990 2562 flags.go:64] FLAG: --runonce="false" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133993 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133996 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.133999 2562 flags.go:64] FLAG: --seccomp-default="false" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134002 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134004 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134007 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134011 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134014 2562 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134017 2562 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134019 2562 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134022 2562 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134025 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134031 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134034 2562 flags.go:64] FLAG: --system-cgroups="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134037 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134043 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134046 2562 flags.go:64] FLAG: --tls-cert-file="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134050 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134054 2562 flags.go:64] FLAG: --tls-min-version="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134058 2562 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134061 2562 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134064 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134067 2562 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134070 2562 flags.go:64] FLAG: --v="2" Nov 03 08:38:55.140223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134074 2562 flags.go:64] FLAG: --version="false" Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134079 2562 flags.go:64] FLAG: --vmodule="" Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134083 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134086 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134246 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134250 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134253 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134257 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134260 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134263 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134267 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134271 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134274 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134277 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134280 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134282 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134285 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134287 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134290 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.140838 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134292 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134296 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134300 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134302 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134305 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134308 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134311 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134314 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134317 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134320 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134322 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134325 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134327 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134330 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134333 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134335 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134338 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134340 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134343 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134346 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.141318 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134348 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134351 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134353 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134356 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134358 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134361 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134363 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134366 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134369 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134371 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134374 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134376 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134379 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134383 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134385 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134391 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134393 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134396 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134398 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134401 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.141808 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134404 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134406 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134409 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134411 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134414 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134416 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134419 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134421 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134424 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134426 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134429 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134431 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134434 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134436 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134439 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134441 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134443 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134446 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134449 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134451 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.142340 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134454 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134456 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134459 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134461 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134464 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134468 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134471 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134473 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134477 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134480 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134482 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134485 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134487 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134490 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134493 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134495 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134499 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134503 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134506 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.142811 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134508 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134511 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134513 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134516 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134518 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134521 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134523 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134526 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134528 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134531 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134533 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134536 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.134565 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134609 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134614 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134617 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.143418 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134620 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134623 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134626 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134630 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134633 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134636 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134639 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134641 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134644 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134647 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134649 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134652 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134654 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134657 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134659 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134662 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134664 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134667 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134669 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.143798 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134672 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134675 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134677 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134680 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134682 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134685 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134687 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134689 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134692 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134695 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134697 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134700 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134702 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134706 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134708 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134711 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134716 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134719 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134722 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134724 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.144321 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134728 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134730 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134733 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134735 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134738 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134740 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134743 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134746 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134748 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134751 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134753 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134756 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134758 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134761 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134763 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134766 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134768 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134771 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134774 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134777 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.144813 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134780 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134782 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134785 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134788 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134790 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134794 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134797 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134800 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134804 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134806 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134809 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134812 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134814 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134817 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134820 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134822 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134825 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134828 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134831 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.145301 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134834 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134837 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134841 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134844 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134846 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134848 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134851 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134854 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134856 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134859 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134862 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134864 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134866 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134869 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134871 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134874 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134876 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134879 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134883 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.145746 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134886 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134888 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134892 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134896 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134901 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.134905 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.142475 2562 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.142494 2562 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.143219 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.143226 2562 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143292 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143298 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143302 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143305 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143308 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143310 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143313 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143316 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143319 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.146213 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143321 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143324 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143326 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143329 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143332 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143334 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143337 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143340 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143342 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143344 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143347 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143349 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143352 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143355 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143357 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143360 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143363 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143365 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143368 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.146663 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143370 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143374 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143377 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143379 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143382 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143385 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143387 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143389 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143392 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143394 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143397 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143399 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143402 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143404 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143406 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143409 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143412 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143414 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143417 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.147118 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143419 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143422 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143431 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143434 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143437 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143439 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143442 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143445 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143447 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143450 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143453 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143456 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143458 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143461 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143464 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143466 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143470 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143472 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143475 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143477 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.147688 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143479 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143482 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143485 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143487 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143490 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143492 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143495 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143497 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143500 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143502 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143505 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143508 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143510 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143513 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143516 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143519 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143522 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143524 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143527 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.148158 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143529 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143532 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143534 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143537 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143540 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143542 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143546 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143551 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143555 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143558 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143563 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143566 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143569 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143571 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143574 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143576 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143579 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143582 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143584 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.148620 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143587 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.143621 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143791 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143797 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143800 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143803 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143807 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143810 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143813 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143815 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143818 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143820 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143824 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143826 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143829 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143832 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.149073 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143835 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143838 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143840 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143843 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143846 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143848 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143851 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143854 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143857 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143860 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143862 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143865 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143867 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143870 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143872 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143875 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143878 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143880 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143883 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.149473 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143886 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143888 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143891 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143893 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143896 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143898 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143901 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143904 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143906 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143909 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143911 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143914 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143917 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143919 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143922 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143924 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143927 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143930 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143934 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143937 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.149947 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143940 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143943 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143946 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143948 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143951 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143954 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143956 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143959 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143961 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143964 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143966 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143969 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143971 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143974 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143976 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143979 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143982 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143985 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143987 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143990 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.150439 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143992 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143995 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.143997 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144001 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144004 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144007 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144010 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144012 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144014 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144017 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144020 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144022 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144025 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144027 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144029 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144032 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144035 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144037 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144040 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.150907 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144043 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144045 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144048 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144050 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144053 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144055 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144057 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144060 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144062 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144065 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144067 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144070 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144072 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144075 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.144097 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144161 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:38:55.151413 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144166 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144170 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144172 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144175 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144177 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144180 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144183 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144186 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144188 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144191 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144194 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144196 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144199 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144202 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144205 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144207 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144209 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144212 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144215 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:38:55.151795 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144218 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144220 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144222 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144225 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144227 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144230 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144232 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144235 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144237 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144240 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144242 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144245 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144248 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144250 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144252 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144255 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144257 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144259 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144262 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:38:55.152237 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144264 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144266 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144269 2562 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144271 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144273 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144276 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144278 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144281 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144283 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144286 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144288 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144291 2562 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144293 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144296 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144298 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144300 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144302 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144306 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144311 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144314 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:38:55.152756 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144317 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144320 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144322 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144325 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144328 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144331 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144334 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144337 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144340 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144342 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144345 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144347 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144349 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144351 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144354 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144357 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144359 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144361 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144364 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144366 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:38:55.153256 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144369 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144372 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144375 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144379 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144382 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144384 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144387 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144389 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144391 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144398 2562 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144401 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144403 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144406 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144408 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144411 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144414 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144417 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144420 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144422 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:38:55.153714 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144425 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144427 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144429 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144432 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144434 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144436 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144439 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:55.144441 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.144500 2562 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.147053 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.148018 2562 server.go:1014] "Starting client certificate rotation" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.148121 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.148184 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.150524 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:38:55.154202 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.151654 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:38:55.163180 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.163156 2562 log.go:25] "Validated CRI v1 runtime API" Nov 03 08:38:55.167404 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.167385 2562 log.go:25] "Validated CRI v1 image API" Nov 03 08:38:55.168316 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.168283 2562 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 08:38:55.172567 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.172538 2562 fs.go:135] Filesystem UUIDs: map[5156017e-e355-44d5-adcc-4ce7e35c91c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0361df5-5b88-4966-948a-a62158183dd8:/dev/nvme0n1p3] Nov 03 08:38:55.172567 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.172563 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 08:38:55.178456 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.178316 2562 manager.go:217] Machine: {Timestamp:2026-11-03 08:38:55.176632314 +0000 UTC m=+0.328303716 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096029 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2170a9ccd7fad3d313fa31d1cc664d SystemUUID:ec2170a9-ccd7-fad3-d313-fa31d1cc664d BootID:c7773d95-afce-483e-935f-bbf241c6f5bc Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:cf:41:b0:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:cf:41:b0:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:00:8f:86:cb:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 08:38:55.179285 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.179271 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 08:38:55.179450 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.179434 2562 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 08:38:55.183520 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183483 2562 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 08:38:55.183677 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183522 2562 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-117.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 08:38:55.183731 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183680 2562 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 08:38:55.183731 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183688 2562 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 08:38:55.183784 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183734 2562 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:38:55.183784 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.183756 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:38:55.184929 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.184916 2562 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 08:38:55.189383 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.189367 2562 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 08:38:55.189525 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.189515 2562 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 08:38:55.190342 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190331 2562 kubelet.go:493] "Attempting to sync node with API server" Nov 03 08:38:55.190374 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190348 2562 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 08:38:55.190374 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190360 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 08:38:55.190374 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190370 2562 kubelet.go:405] "Adding apiserver pod source" Nov 03 08:38:55.190473 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190380 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 08:38:55.190729 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.190717 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:38:55.193405 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.193387 2562 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 08:38:55.200465 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.200443 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:38:55.201348 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201330 2562 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 08:38:55.201424 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201372 2562 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 08:38:55.201596 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201581 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 08:38:55.201657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201604 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 08:38:55.201657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201614 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 08:38:55.201657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201623 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 08:38:55.201657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201632 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 08:38:55.201657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.201641 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 08:38:55.202291 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202279 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 08:38:55.202342 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202295 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 08:38:55.202342 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202308 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 08:38:55.202342 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202319 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 08:38:55.202342 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202334 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 08:38:55.202484 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.202348 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 08:38:55.203593 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.203581 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 08:38:55.203646 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.203597 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 08:38:55.207600 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.207582 2562 server.go:1263] "Started kubelet" Nov 03 08:38:55.207717 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.207687 2562 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 08:38:55.207791 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.207740 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 08:38:55.207828 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.207817 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 08:38:55.209170 ip-10-0-131-117 systemd[1]: Started Kubernetes Kubelet. Nov 03 08:38:55.209322 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.209239 2562 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 08:38:55.211760 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.211743 2562 server.go:317] "Adding debug handlers to kubelet server" Nov 03 08:38:55.213922 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.213905 2562 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 08:38:55.216036 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.216020 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 08:38:55.216538 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.216519 2562 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 08:38:55.216891 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.216723 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:55.217281 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217261 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 08:38:55.217281 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217267 2562 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 08:38:55.217568 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217289 2562 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 08:38:55.217568 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217377 2562 reconstruct.go:97] "Volume reconstruction finished" Nov 03 08:38:55.217568 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217384 2562 reconciler.go:29] "Reconciler: start to sync state" Nov 03 08:38:55.217568 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217586 2562 factory.go:153] Registering CRI-O factory Nov 03 08:38:55.217759 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217599 2562 factory.go:223] Registration of the crio container factory successfully Nov 03 08:38:55.217837 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217821 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 08:38:55.217890 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217839 2562 factory.go:55] Registering systemd factory Nov 03 08:38:55.217890 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217847 2562 factory.go:223] Registration of the systemd container factory successfully Nov 03 08:38:55.217890 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217865 2562 factory.go:103] Registering Raw factory Nov 03 08:38:55.218035 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.217966 2562 manager.go:1199] Started watching for new ooms in manager Nov 03 08:38:55.219040 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.219018 2562 manager.go:319] Starting recovery of all containers Nov 03 08:38:55.229833 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.229809 2562 manager.go:324] Recovery completed Nov 03 08:38:55.231043 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.231010 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 03 08:38:55.233972 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.233957 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.235596 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.235569 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.235657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.235615 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.235657 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.235626 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.236178 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236162 2562 cpu_manager.go:226] "Starting" policy="none" Nov 03 08:38:55.236178 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236177 2562 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 08:38:55.236289 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236195 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.236289 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236199 2562 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 08:38:55.236888 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236873 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.236950 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236900 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.236950 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.236910 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.238481 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.238468 2562 policy_none.go:50] "Start" Nov 03 08:38:55.238531 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.238485 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 08:38:55.238531 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.238496 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 08:38:55.240584 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.240567 2562 policy_none.go:44] "Start" Nov 03 08:38:55.272035 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.271866 2562 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.284468 2562 manager.go:341] "Starting Device Plugin manager" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.284509 2562 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.284522 2562 server.go:87] "Starting device plugin registration server" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.284797 2562 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.284810 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.284907 2562 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.285000 2562 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.285007 2562 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.285721 2562 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.285787 2562 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.296830 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.298236 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.298256 2562 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.298275 2562 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 08:38:55.303615 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.298304 2562 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 08:38:55.350443 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.350396 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-117.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 08:38:55.351481 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.350402 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-117.us-west-1.compute.internal.18e47c38d96471b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-117.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-117.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55.207551412 +0000 UTC m=+0.359222815,LastTimestamp:2026-11-03 08:38:55.207551412 +0000 UTC m=+0.359222815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-117.us-west-1.compute.internal,}" Nov 03 08:38:55.385559 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.385489 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.386437 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.386419 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.386551 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.386458 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.386551 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.386483 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.386551 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.386516 2562 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.399010 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.398982 2562 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal"] Nov 03 08:38:55.399119 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.399009 2562 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-117.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 08:38:55.399119 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.399047 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.399947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.399930 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.400022 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.399960 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.400022 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.399973 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.402120 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402109 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.402276 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402262 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.402327 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402292 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.402892 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402871 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.403004 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402908 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.403004 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402910 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.403004 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402925 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.403004 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402936 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.403004 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.402949 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.405063 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.405049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.405128 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.405075 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.406203 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.406180 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.406277 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.406209 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.406277 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.406219 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.443754 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.443728 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.445732 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.445713 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.456932 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.456901 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-117.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 08:38:55.456932 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.456911 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 08:38:55.457119 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.456882 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-117.us-west-1.compute.internal.18e47c38db10698d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-117.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-117.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-117.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55.235598733 +0000 UTC m=+0.387270135,LastTimestamp:2026-11-03 08:38:55.235598733 +0000 UTC m=+0.387270135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-117.us-west-1.compute.internal,}" Nov 03 08:38:55.457266 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.457233 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 08:38:55.457362 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.457338 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 08:38:55.476656 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.476637 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 08:38:55.476772 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.476686 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l5b74" Nov 03 08:38:55.512363 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.510526 2562 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-117.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.512363 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.511082 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-117.us-west-1.compute.internal.18e47c38db10beac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-117.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-117.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-117.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55.235620524 +0000 UTC m=+0.387291926,LastTimestamp:2026-11-03 08:38:55.235620524 +0000 UTC m=+0.387291926,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-117.us-west-1.compute.internal,}" Nov 03 08:38:55.518175 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.518153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.518256 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.518181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.518256 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.518199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618795 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618795 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618797 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618989 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618989 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618877 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618989 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618879 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.618989 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.618916 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.707846 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.707768 2562 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-117.us-west-1.compute.internal\" not found" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.710866 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.710850 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:55.711832 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.711817 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:55.711926 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.711844 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:55.711926 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.711856 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:55.711926 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.711882 2562 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.748382 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.748356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.752310 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.752290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.768284 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:55.768259 2562 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:55.768284 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.768285 2562 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-117.us-west-1.compute.internal\": node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:55.887171 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.887125 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:55.988057 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:55.987958 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.088386 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.088352 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.148434 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.148389 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 08:38:56.148865 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.148564 2562 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aaf99b2950bda4e3eb23748958ae1964-ce6ebf1c3713135a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-117.us-west-1.compute.internal.18e47c38db10698d\": read tcp 10.0.131.117:57878->3.130.62.217:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-117.us-west-1.compute.internal.18e47c38db10698d default 5157 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-117.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-117.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-117.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55 +0000 UTC,LastTimestamp:2026-11-03 08:38:55.39994649 +0000 UTC m=+0.551617892,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-117.us-west-1.compute.internal,}" Nov 03 08:38:56.188713 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.188688 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.289672 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.289593 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.390706 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.390655 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.482340 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.482282 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:56 +0000 UTC" deadline="2028-08-15 14:06:08.472363868 +0000 UTC" Nov 03 08:38:56.482340 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.482326 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15629h27m11.990040804s" Nov 03 08:38:56.491457 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:56.491432 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:38:56.510815 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.510791 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:38:56.511781 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.511765 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:38:56.511888 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.511792 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:38:56.511888 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.511802 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:38:56.576230 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.576148 2562 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:56.617469 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.617419 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:56.627387 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.627358 2562 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:56.627571 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.627554 2562 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:56.678959 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.678926 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:38:56.679892 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.679876 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:38:56.740904 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.740881 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:38:56.846570 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:56.846523 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050f26e6d5e63142521b2810eac72bf7.slice/crio-988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748 WatchSource:0}: Error finding container 988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748: Status 404 returned error can't find the container with id 988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748 Nov 03 08:38:56.846986 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:56.846962 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fdfbfab5aa17ccf7324ae585a08734.slice/crio-16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4 WatchSource:0}: Error finding container 16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4: Status 404 returned error can't find the container with id 16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4 Nov 03 08:38:56.848769 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.848745 2562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:38:56.852021 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:56.852007 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:38:57.192193 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.192155 2562 apiserver.go:52] "Watching apiserver" Nov 03 08:38:57.217711 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.217668 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 08:38:57.303510 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.303452 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"20fdfbfab5aa17ccf7324ae585a08734","Type":"ContainerStarted","Data":"16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4"} Nov 03 08:38:57.304528 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.304493 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerStarted","Data":"988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748"} Nov 03 08:38:57.306438 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.306419 2562 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 08:38:57.306772 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.306755 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-n26bm","kube-system/konnectivity-agent-9rkp2","openshift-multus/multus-additional-cni-plugins-bj87v","openshift-multus/network-metrics-daemon-d8jcr","openshift-network-diagnostics/network-check-target-w4gf5","openshift-network-operator/iptables-alerter-62fp4","kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal","openshift-dns/node-resolver-6pr47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal","openshift-multus/multus-4xgnk","openshift-cluster-node-tuning-operator/tuned-f4ldt","kube-system/global-pull-secret-syncer-qkzjz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j","openshift-image-registry/node-ca-dfmxv"] Nov 03 08:38:57.310815 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.310796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.313035 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.313015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.315306 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.315288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.318826 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.318804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.318930 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.318892 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:38:57.321104 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.321087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:57.321216 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.321160 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:38:57.323471 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.323450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.325843 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.325826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.328015 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.327996 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328109 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328109 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328109 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328287 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.328287 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z86ww\" (UniqueName: \"kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.328287 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328287 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328265 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328317 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.328488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328488 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328677 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.328779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvxg\" (UniqueName: \"kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncrg\" (UniqueName: \"kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnlg\" (UniqueName: \"kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.328976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.329262 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.329014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.329469 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.329362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.331860 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.331843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.334590 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.334556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.337372 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.337351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.339722 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.339706 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.339782 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.339758 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:38:57.370006 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.369987 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\"" Nov 03 08:38:57.370123 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370073 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.370123 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370119 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 08:38:57.370243 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370128 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 08:38:57.370376 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370360 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.370424 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370407 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.370650 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370632 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.370765 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370646 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 08:38:57.370883 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.370868 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\"" Nov 03 08:38:57.372595 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.372583 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 08:38:57.372669 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.372652 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 08:38:57.372790 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.372779 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\"" Nov 03 08:38:57.418996 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.418934 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 08:38:57.429915 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.429894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.430018 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.429928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.430018 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.429945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:57.430018 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.429961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430018 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.429975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.430184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.430184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430355 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.430355 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npcm\" (UniqueName: \"kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.430355 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430355 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.430355 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430331 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.430565 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430545 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430506 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430572 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430681 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430724 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430750 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430808 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.430848 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430924 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.430990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431039 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431080 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431099 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431120 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431183 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.431305 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2f5\" (UniqueName: \"kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvxg\" (UniqueName: \"kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431449 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hl8n\" (UniqueName: \"kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431532 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431605 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431625 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnlg\" (UniqueName: \"kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctsb\" (UniqueName: \"kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431770 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431818 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.431988 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.431974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86ww\" (UniqueName: \"kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432036 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432192 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.432334 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.432474 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:57.932414635 +0000 UTC m=+3.084086045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432494 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.432793 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432635 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432634 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432668 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432767 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432796 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432907 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432980 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.432988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c752x\" (UniqueName: \"kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.433449 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433053 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433062 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433127 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncrg\" (UniqueName: \"kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.433962 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.433569 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.478858 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.478816 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.479377 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479350 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.479377 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479369 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479411 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479438 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479465 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479509 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479507 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.479526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479532 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479587 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479726 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479738 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479740 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479728 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\"" Nov 03 08:38:57.479819 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479798 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479974 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.479998 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480003 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480035 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480043 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480055 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480052 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480112 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 08:38:57.480168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.480135 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 08:38:57.493244 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.493213 2562 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 08:38:57.497008 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.496983 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvxg\" (UniqueName: \"kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.497173 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.496994 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncrg\" (UniqueName: \"kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.497173 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.497035 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnlg\" (UniqueName: \"kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.497173 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.497035 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86ww\" (UniqueName: \"kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.533650 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.533650 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533726 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533790 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533834 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533885 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533896 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533887 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533913 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.533932 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533933 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.533933 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533973 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.533938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.534010 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.033988681 +0000 UTC m=+3.185660075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c752x\" (UniqueName: \"kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534161 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534196 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534228 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.534583 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534377 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534486 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npcm\" (UniqueName: \"kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534601 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534648 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534673 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534695 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534843 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534843 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534896 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.535493 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.534994 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535023 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535073 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535113 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535218 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535213 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2f5\" (UniqueName: \"kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536359 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535465 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535482 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl8n\" (UniqueName: \"kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535539 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctsb\" (UniqueName: \"kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535602 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535641 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535539 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535828 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535873 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.536947 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535925 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.535965 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.536052 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.536282 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.537196 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.537506 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.537298 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.537852 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.537829 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.538176 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.538159 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.546841 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.546816 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 08:38:57.546954 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.546908 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7nfb9" Nov 03 08:38:57.596510 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.596473 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:57.596510 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.596503 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:57.596510 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.596518 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:57.596745 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.596596 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.096576784 +0000 UTC m=+3.248248190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:57.598823 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.598784 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c752x\" (UniqueName: \"kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.599361 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.599340 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npcm\" (UniqueName: \"kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.599487 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.599465 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2f5\" (UniqueName: \"kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.599619 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.599600 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl8n\" (UniqueName: \"kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.599713 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.599694 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctsb\" (UniqueName: \"kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.624510 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.624473 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4xgnk" Nov 03 08:38:57.634381 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.634350 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518c5a4f_9809_452d_8a8d_aea90d5961c5.slice/crio-1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1 WatchSource:0}: Error finding container 1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1: Status 404 returned error can't find the container with id 1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1 Nov 03 08:38:57.635533 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.635513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:38:57.643747 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.643723 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c823c7a_c6fb_4796_9a85_5f23e6c05759.slice/crio-94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59 WatchSource:0}: Error finding container 94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59: Status 404 returned error can't find the container with id 94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59 Nov 03 08:38:57.652309 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.652284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:38:57.652412 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.652367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:38:57.656331 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.656311 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:38:57.658396 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.658382 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pr47" Nov 03 08:38:57.661604 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.661572 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530f4286_b8c2_4331_9bcc_372807e290d9.slice/crio-284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8 WatchSource:0}: Error finding container 284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8: Status 404 returned error can't find the container with id 284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8 Nov 03 08:38:57.661705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.661673 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:38:57.667675 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.667648 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57c4f2b_51b2_43d1_9fa4_052405923764.slice/crio-f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3 WatchSource:0}: Error finding container f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3: Status 404 returned error can't find the container with id f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3 Nov 03 08:38:57.668227 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.668203 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fbcb5f_5b60_488c_929a_e5661647fb3a.slice/crio-cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047 WatchSource:0}: Error finding container cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047: Status 404 returned error can't find the container with id cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047 Nov 03 08:38:57.672422 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.672340 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f00260d_22ef_44e7_802c_d1b9b9585b61.slice/crio-8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13 WatchSource:0}: Error finding container 8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13: Status 404 returned error can't find the container with id 8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13 Nov 03 08:38:57.674200 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.674171 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0527392_1028_47e0_a542_80038e79f6da.slice/crio-9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19 WatchSource:0}: Error finding container 9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19: Status 404 returned error can't find the container with id 9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19 Nov 03 08:38:57.678417 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.678397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:38:57.686041 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.686020 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:38:57.689091 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.689067 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8977ebc_ddf6_41df_9faf_175c92054114.slice/crio-4134b6749ef5397f39b75bba38c78cf62bb22ed0a6bba0f49184ec06888d7ab8 WatchSource:0}: Error finding container 4134b6749ef5397f39b75bba38c78cf62bb22ed0a6bba0f49184ec06888d7ab8: Status 404 returned error can't find the container with id 4134b6749ef5397f39b75bba38c78cf62bb22ed0a6bba0f49184ec06888d7ab8 Nov 03 08:38:57.695181 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:38:57.695161 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e5037f_1bbd_4170_8108_6a06e3f782f2.slice/crio-6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1 WatchSource:0}: Error finding container 6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1: Status 404 returned error can't find the container with id 6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1 Nov 03 08:38:57.938113 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:57.938028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:57.938282 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.938186 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:57.938282 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:57.938258 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.938230079 +0000 UTC m=+4.089901484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:58.038760 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.038718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:58.038940 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.038901 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:58.038999 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.038969 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:38:59.038948727 +0000 UTC m=+4.190620139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:58.139484 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.139446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:58.139683 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.139632 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:58.139683 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.139658 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:58.139683 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.139671 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:58.139858 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.139736 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:38:59.139716708 +0000 UTC m=+4.291388114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:58.308380 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.308267 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" event={"ID":"8c823c7a-c6fb-4796-9a85-5f23e6c05759","Type":"ContainerStarted","Data":"94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59"} Nov 03 08:38:58.309785 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.309756 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4xgnk" event={"ID":"518c5a4f-9809-452d-8a8d-aea90d5961c5","Type":"ContainerStarted","Data":"1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1"} Nov 03 08:38:58.311286 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.311256 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1"} Nov 03 08:38:58.312578 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.312549 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" event={"ID":"b8977ebc-ddf6-41df-9faf-175c92054114","Type":"ContainerStarted","Data":"4134b6749ef5397f39b75bba38c78cf62bb22ed0a6bba0f49184ec06888d7ab8"} Nov 03 08:38:58.314124 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.314099 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rkp2" event={"ID":"e0527392-1028-47e0-a542-80038e79f6da","Type":"ContainerStarted","Data":"9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19"} Nov 03 08:38:58.315254 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.315234 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pr47" event={"ID":"2f00260d-22ef-44e7-802c-d1b9b9585b61","Type":"ContainerStarted","Data":"8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13"} Nov 03 08:38:58.316537 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.316514 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerStarted","Data":"cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047"} Nov 03 08:38:58.317739 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.317720 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dfmxv" event={"ID":"a57c4f2b-51b2-43d1-9fa4-052405923764","Type":"ContainerStarted","Data":"f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3"} Nov 03 08:38:58.318933 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.318895 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-62fp4" event={"ID":"530f4286-b8c2-4331-9bcc-372807e290d9","Type":"ContainerStarted","Data":"284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8"} Nov 03 08:38:58.551440 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.551369 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:58 +0000 UTC" deadline="2028-03-29 12:43:06.951980662 +0000 UTC" Nov 03 08:38:58.551440 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.551437 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12292h4m8.400548002s" Nov 03 08:38:58.946463 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:58.946188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:58.946463 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.946362 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:58.946463 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:58.946427 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:00.94640686 +0000 UTC m=+6.098078255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:38:59.047815 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.047556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:59.047815 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.047709 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:59.047815 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.047776 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.047754913 +0000 UTC m=+6.199426326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:38:59.148803 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.148759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:59.148980 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.148960 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:38:59.149059 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.148992 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:38:59.149059 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.149006 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:59.149176 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.149070 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.149050835 +0000 UTC m=+6.300722226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:38:59.299452 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.299415 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:38:59.299648 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.299592 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:38:59.299725 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.299709 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:38:59.299879 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.299857 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:38:59.301361 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.301337 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:38:59.301485 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:38:59.301437 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:38:59.552508 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.552385 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:58 +0000 UTC" deadline="2028-04-16 10:46:40.936624957 +0000 UTC" Nov 03 08:38:59.552508 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:38:59.552428 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12722h7m41.384201683s" Nov 03 08:39:00.966690 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:00.966438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:00.967163 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:00.966843 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:00.967163 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:00.966905 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:04.966887084 +0000 UTC m=+10.118558480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:01.068435 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.067757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:01.068435 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.067906 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:01.068435 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.067960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:39:05.067947512 +0000 UTC m=+10.219618902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:01.169704 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.169278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:01.169704 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.169466 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:39:01.169704 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.169493 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:39:01.169704 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.169508 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:01.169704 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.169572 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:05.169553258 +0000 UTC m=+10.321224672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:01.300907 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.300873 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:01.301080 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.300988 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:01.301391 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.301360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:01.301527 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.301399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:01.301527 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.301506 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:01.301631 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:01.301572 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:01.360244 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:01.359412 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"20fdfbfab5aa17ccf7324ae585a08734","Type":"ContainerStarted","Data":"1c18fbe5a3943ccfc1839901cc0ed5a866678bb9ba401ad9bdf785ba325232d2"} Nov 03 08:39:02.368128 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:02.368085 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="049c4c36c3c08c753488e6d0448f445e93e226ff2ffb64ddbe80a44d4062d300" exitCode=0 Nov 03 08:39:02.368661 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:02.368196 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"049c4c36c3c08c753488e6d0448f445e93e226ff2ffb64ddbe80a44d4062d300"} Nov 03 08:39:02.377794 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:02.376682 2562 generic.go:356] "Generic (PLEG): container finished" podID="050f26e6d5e63142521b2810eac72bf7" containerID="2e5874725d3854eea58c260613abe6491202477efb7f5704fd290a3f944591c4" exitCode=0 Nov 03 08:39:02.377794 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:02.377606 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerDied","Data":"2e5874725d3854eea58c260613abe6491202477efb7f5704fd290a3f944591c4"} Nov 03 08:39:02.592769 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:02.592688 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" podStartSLOduration=5.5926647240000005 podStartE2EDuration="5.592664724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:01.487337376 +0000 UTC m=+6.639008792" watchObservedRunningTime="2026-11-03 08:39:02.592664724 +0000 UTC m=+7.744336139" Nov 03 08:39:03.300392 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.300358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:03.300580 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:03.300518 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:03.300986 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.300959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:03.301104 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:03.301078 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:03.301209 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.301162 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:03.301310 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:03.301288 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:03.396207 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.396166 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rkp2" event={"ID":"e0527392-1028-47e0-a542-80038e79f6da","Type":"ContainerStarted","Data":"5ed1e7f8277cc4b99d5d50c14d10d38fabcf19f8adde271f4e17aaf959a8214c"} Nov 03 08:39:03.401730 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.401701 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerStarted","Data":"8905cb440dd789eac72ee447160282a4784517333d75ec9c733605dc4cb5ea8c"} Nov 03 08:39:03.653066 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.652999 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rkp2" podStartSLOduration=2.7271476740000002 podStartE2EDuration="7.652976931s" totalImagesPullingTime="4.925829257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:03.528437286 +0000 UTC m=+8.680108701" watchObservedRunningTime="2026-11-03 08:39:03.652976931 +0000 UTC m=+8.804648347" Nov 03 08:39:03.653981 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:03.653510 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" podStartSLOduration=6.653495426 podStartE2EDuration="6.653495426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:03.652787305 +0000 UTC m=+8.804458721" watchObservedRunningTime="2026-11-03 08:39:03.653495426 +0000 UTC m=+8.805166840" Nov 03 08:39:04.407339 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:04.407298 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dfmxv" event={"ID":"a57c4f2b-51b2-43d1-9fa4-052405923764","Type":"ContainerStarted","Data":"204718e49e971aa051e5d602d97a46f3aedd179fa30359ecfdf7a939a23dd190"} Nov 03 08:39:05.006645 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.006606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:05.006844 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.006820 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:05.006934 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.006888 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.00687038 +0000 UTC m=+18.158541786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:05.107063 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.107014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:05.107288 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.107252 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:05.107364 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.107317 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.107299952 +0000 UTC m=+18.258971364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:05.208347 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.208307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:05.208519 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.208485 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:39:05.208519 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.208508 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:39:05.208634 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.208521 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:05.208634 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.208584 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.208565212 +0000 UTC m=+18.360236619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:05.228426 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.228394 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:05.228920 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.228893 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:05.300548 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.300477 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:05.300718 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.300603 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:05.301348 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.300954 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:05.301348 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.301063 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:05.301348 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.301130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:05.301348 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:05.301197 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:05.358753 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.356589 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dfmxv" podStartSLOduration=3.325484268 podStartE2EDuration="9.356567953s" totalImagesPullingTime="6.031083685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:04.533109719 +0000 UTC m=+9.684781135" watchObservedRunningTime="2026-11-03 08:39:05.356567953 +0000 UTC m=+10.508239370" Nov 03 08:39:05.413129 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.412438 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" event={"ID":"8c823c7a-c6fb-4796-9a85-5f23e6c05759","Type":"ContainerStarted","Data":"9172d90841094bb1252b384f321791a3d16b8c076cfbfec61d8ef591f9ea1781"} Nov 03 08:39:05.413129 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.412507 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:05.414104 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:05.413890 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:06.575366 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:06.575331 2562 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 08:39:07.292128 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.291981 2562 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:39:06.575362515Z","UUID":"175915ae-c2fc-45be-8be0-cb151d4b2741","Handler":null,"Name":"","Endpoint":""} Nov 03 08:39:07.293831 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.293806 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 08:39:07.293952 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.293838 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 08:39:07.299205 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.299180 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:07.299304 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.299182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:07.299414 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:07.299316 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:07.299414 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.299191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:07.299414 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:07.299357 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:07.299576 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:07.299423 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:07.419275 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.419240 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" event={"ID":"8c823c7a-c6fb-4796-9a85-5f23e6c05759","Type":"ContainerStarted","Data":"0fa8f3860ba9334c147f7081ea1d03fa4ed66d04d81f2a793b7472c1752ddfd2"} Nov 03 08:39:07.421574 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.421543 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4xgnk" event={"ID":"518c5a4f-9809-452d-8a8d-aea90d5961c5","Type":"ContainerStarted","Data":"a25ef51ec9cbc2c7bda8f0b551e65f21432ffa54ff0bd83c8842dcfb10f27cfb"} Nov 03 08:39:07.423393 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.423365 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pr47" event={"ID":"2f00260d-22ef-44e7-802c-d1b9b9585b61","Type":"ContainerStarted","Data":"cd351663b9130fa3eb9eb0f6d8eb85c9baa1490e74da1e44d674b487d8c2fc31"} Nov 03 08:39:07.652800 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:07.652590 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4xgnk" podStartSLOduration=2.687631262 podStartE2EDuration="11.652576421s" totalImagesPullingTime="8.964945159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:07.538089301 +0000 UTC m=+12.689760711" watchObservedRunningTime="2026-11-03 08:39:07.652576421 +0000 UTC m=+12.804247832" Nov 03 08:39:08.427555 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.427510 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="723aed93d1e6461be60428cbb04051bf417a8272a581ce6b55cb52bf5598bca1" exitCode=0 Nov 03 08:39:08.427762 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.427605 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"723aed93d1e6461be60428cbb04051bf417a8272a581ce6b55cb52bf5598bca1"} Nov 03 08:39:08.431423 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.431397 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-62fp4" event={"ID":"530f4286-b8c2-4331-9bcc-372807e290d9","Type":"ContainerStarted","Data":"73b73730bccbfa76f6540cd888b066fe8412008aab55ce35d78156e64c1816cf"} Nov 03 08:39:08.434480 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.434453 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" event={"ID":"8c823c7a-c6fb-4796-9a85-5f23e6c05759","Type":"ContainerStarted","Data":"ff7cacc90fb584bcc9bb10bd3327d1c34339532ce55530d16265ca37c3394c1b"} Nov 03 08:39:08.547036 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.546990 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6pr47" podStartSLOduration=3.289535229 podStartE2EDuration="12.546978325s" totalImagesPullingTime="9.257443096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:07.653326115 +0000 UTC m=+12.804997527" watchObservedRunningTime="2026-11-03 08:39:08.546978325 +0000 UTC m=+13.698649736" Nov 03 08:39:08.663915 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.663869 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" podStartSLOduration=2.688208081 podStartE2EDuration="12.663851946s" totalImagesPullingTime="9.975643865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:08.663740125 +0000 UTC m=+13.815411538" watchObservedRunningTime="2026-11-03 08:39:08.663851946 +0000 UTC m=+13.815523360" Nov 03 08:39:08.779887 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:08.779798 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-62fp4" podStartSLOduration=3.848916463 podStartE2EDuration="12.77978419s" totalImagesPullingTime="8.930867727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:08.779282729 +0000 UTC m=+13.930954140" watchObservedRunningTime="2026-11-03 08:39:08.77978419 +0000 UTC m=+13.931455601" Nov 03 08:39:09.299357 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:09.299332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:09.299484 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:09.299461 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:09.299544 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:09.299510 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:09.299596 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:09.299555 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:09.299596 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:09.299590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:09.299688 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:09.299649 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:09.437884 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:09.437833 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" event={"ID":"b8977ebc-ddf6-41df-9faf-175c92054114","Type":"ContainerStarted","Data":"b4c477022b4ef911435c906d77625786709add6559657257ac92e0efc500467b"} Nov 03 08:39:09.555177 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:09.555063 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" podStartSLOduration=2.036268129 podStartE2EDuration="13.555048643s" totalImagesPullingTime="11.518780514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:09.554476441 +0000 UTC m=+14.706147854" watchObservedRunningTime="2026-11-03 08:39:09.555048643 +0000 UTC m=+14.706720058" Nov 03 08:39:10.441787 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:10.441747 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="39e4bc03e41e2280522881311b72eaa073d3aba534f6698f0df45566bf71f59f" exitCode=0 Nov 03 08:39:10.442567 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:10.441838 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"39e4bc03e41e2280522881311b72eaa073d3aba534f6698f0df45566bf71f59f"} Nov 03 08:39:11.298643 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:11.298603 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:11.298643 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:11.298629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:11.298869 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:11.298635 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:11.298869 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:11.298732 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:11.298869 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:11.298839 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:11.299010 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:11.298899 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:12.448706 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:12.448487 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="bd4f745079caf45b72868108abfcd29843d49e522034a1aa853f7de4458b2ee0" exitCode=0 Nov 03 08:39:12.448706 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:12.448571 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"bd4f745079caf45b72868108abfcd29843d49e522034a1aa853f7de4458b2ee0"} Nov 03 08:39:13.067107 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.067077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:13.067303 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.067193 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:13.067303 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.067243 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.067229872 +0000 UTC m=+34.218901265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 08:39:13.167465 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.167437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:13.167597 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.167530 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:13.167597 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.167574 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret podName:f61e725e-6128-4977-9ac4-ee0df1feb8db nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.167561443 +0000 UTC m=+34.319232835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret") pod "global-pull-secret-syncer-qkzjz" (UID: "f61e725e-6128-4977-9ac4-ee0df1feb8db") : object "kube-system"/"original-pull-secret" not registered Nov 03 08:39:13.268672 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.268645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:13.268759 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.268751 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 08:39:13.268795 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.268764 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 08:39:13.268795 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.268772 2562 projected.go:196] Error preparing data for projected volume kube-api-access-jkkn2 for pod openshift-network-diagnostics/network-check-target-w4gf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:13.268852 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.268812 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2 podName:7d926adb-98d1-46bb-a1e9-8624624c0f2f nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.268801221 +0000 UTC m=+34.420472611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jkkn2" (UniqueName: "kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2") pod "network-check-target-w4gf5" (UID: "7d926adb-98d1-46bb-a1e9-8624624c0f2f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 08:39:13.298930 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.298896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:13.299014 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.298903 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:13.299079 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.299038 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:13.299079 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.298903 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:13.299207 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.299095 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:13.299207 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:13.299161 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:13.454297 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.454276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:13.454688 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.454665 2562 generic.go:356] "Generic (PLEG): container finished" podID="27e5037f-1bbd-4170-8108-6a06e3f782f2" containerID="c5a38d5da4bd4d26b50666b5c3b0ca99ac2d0bd3e7d55a12eba7b930d5ef17e2" exitCode=1 Nov 03 08:39:13.454753 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.454732 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerDied","Data":"c5a38d5da4bd4d26b50666b5c3b0ca99ac2d0bd3e7d55a12eba7b930d5ef17e2"} Nov 03 08:39:13.454822 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:13.454808 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"6c0a95a046de31ebf7ad87d92e90662d6b25ee60173d1f9227e82d614922b630"} Nov 03 08:39:14.459695 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.459470 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerStarted","Data":"44db2e456eaab305387eb9e73c44ccabd839b608371e12436754c704353e74c5"} Nov 03 08:39:14.462460 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.462441 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:14.462841 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.462816 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"d7a9303001e0ddc435da2453f8b2ca2582881d6d96f84ec295313581cec6126b"} Nov 03 08:39:14.462919 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.462903 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"d44f675306eab0465c831861bec8ff41ee76adb9ea051af29cc4a807d5656407"} Nov 03 08:39:14.462973 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.462961 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"b5737e410737656e1210b4dece24acacff61917d9b1f4af044e458bcc378f408"} Nov 03 08:39:14.463029 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:14.463013 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"91640d06e94932119936cf5d40a5d9ebc151f2be29bb1cd64586486f3e109228"} Nov 03 08:39:15.299544 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.299511 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:15.299705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.299618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:15.299705 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:15.299633 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:15.299705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.299660 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:15.299890 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:15.299736 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:15.299890 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:15.299816 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:15.466899 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.466866 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="44db2e456eaab305387eb9e73c44ccabd839b608371e12436754c704353e74c5" exitCode=0 Nov 03 08:39:15.466899 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.466901 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"44db2e456eaab305387eb9e73c44ccabd839b608371e12436754c704353e74c5"} Nov 03 08:39:15.835526 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:15.835452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:16.022403 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.022376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:16.218838 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.218813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:16.348680 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.348652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:16.471286 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.471232 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:16.471640 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.471595 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"4ce9644e11c90f495d7674b45df8ae9a85823ad9c2d07936baed79b211c1d6ad"} Nov 03 08:39:16.474389 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.474366 2562 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="878eeba4410bed3b2b74897062ecf4f58b2fc2b1d4c9371f1ee8471fc088a9d9" exitCode=0 Nov 03 08:39:16.474489 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:16.474409 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerDied","Data":"878eeba4410bed3b2b74897062ecf4f58b2fc2b1d4c9371f1ee8471fc088a9d9"} Nov 03 08:39:17.071812 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.071777 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:17.263614 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.263585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:17.299402 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.299381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:17.299488 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.299402 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:17.299488 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:17.299465 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:17.299559 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.299509 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:17.299644 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:17.299621 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:17.299719 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:17.299702 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:17.479707 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.479679 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bj87v" event={"ID":"15fbcb5f-5b60-488c-929a-e5661647fb3a","Type":"ContainerStarted","Data":"1770b576da75859228676c3032d1b685781211e8eef44ebe1cfc933020a480e3"} Nov 03 08:39:17.603555 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:17.603512 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bj87v" podStartSLOduration=8.640900111 podStartE2EDuration="21.603498715s" totalImagesPullingTime="12.962598604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:17.602863899 +0000 UTC m=+22.754535310" watchObservedRunningTime="2026-11-03 08:39:17.603498715 +0000 UTC m=+22.755170145" Nov 03 08:39:18.487325 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:18.487168 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:18.487868 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:18.487624 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"e1d3cfcb7d7168409d321d5e18b2614965042cea4b3a97207a1f5db885398715"} Nov 03 08:39:18.488194 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:18.488179 2562 scope.go:122] "RemoveContainer" containerID="c5a38d5da4bd4d26b50666b5c3b0ca99ac2d0bd3e7d55a12eba7b930d5ef17e2" Nov 03 08:39:19.299333 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.299294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:19.299333 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.299328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:19.299596 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:19.299416 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:19.299596 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:19.299458 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:19.299596 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.299481 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:19.299596 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:19.299536 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:19.492865 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.492833 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:19.493293 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.493209 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" event={"ID":"27e5037f-1bbd-4170-8108-6a06e3f782f2","Type":"ContainerStarted","Data":"4052dd322df59dd0114e94d0f15561b2f51c3b60953d767694dbace3506ee523"} Nov 03 08:39:19.493428 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.493414 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:19.493514 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.493503 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:19.493563 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.493548 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:19.508475 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.508453 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:19.508779 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.508763 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:19.728500 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.728462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:19.741346 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.741301 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" podStartSLOduration=8.213255262 podStartE2EDuration="23.741289529s" totalImagesPullingTime="15.528034267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:19.74085515 +0000 UTC m=+24.892526561" watchObservedRunningTime="2026-11-03 08:39:19.741289529 +0000 UTC m=+24.892960941" Nov 03 08:39:19.859952 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:19.859920 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:20.480592 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.480513 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8jcr"] Nov 03 08:39:20.480748 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.480657 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:20.480809 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:20.480748 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:20.481057 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.481030 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkzjz"] Nov 03 08:39:20.481229 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.481156 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:20.481300 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:20.481250 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:20.482662 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.482638 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w4gf5"] Nov 03 08:39:20.482762 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:20.482741 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:20.482847 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:20.482829 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:22.298680 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:22.298644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:22.299110 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:22.298685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:22.299110 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:22.298685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:22.299110 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:22.298784 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:22.299110 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:22.298871 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:22.299110 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:22.299047 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:24.118731 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.118694 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:24.253980 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.253949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:24.298839 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.298809 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:24.298839 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.298838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:24.299000 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:24.298896 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkzjz" podUID="f61e725e-6128-4977-9ac4-ee0df1feb8db" Nov 03 08:39:24.299000 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.298814 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:24.299000 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:24.298976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w4gf5" podUID="7d926adb-98d1-46bb-a1e9-8624624c0f2f" Nov 03 08:39:24.299089 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:24.299051 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d8jcr" podUID="bec20a64-e1e9-49fc-bb02-d62c451121e1" Nov 03 08:39:24.438857 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.438796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:24.573269 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.573241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:24.795830 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.795769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:24.953367 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:24.953338 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:25.089873 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.089792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:25.493545 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.493391 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeReady" Nov 03 08:39:25.493884 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.493637 2562 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 08:39:25.646906 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.646877 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-45lmv"] Nov 03 08:39:25.648643 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.648629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.658389 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.658368 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pd45j"] Nov 03 08:39:25.660109 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.660095 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-678m8"] Nov 03 08:39:25.660255 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.660237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.661605 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.661583 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.669204 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.669178 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd45j"] Nov 03 08:39:25.670052 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.670032 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45lmv"] Nov 03 08:39:25.702129 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.702103 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 08:39:25.702129 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.702123 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\"" Nov 03 08:39:25.703637 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.703622 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 08:39:25.713316 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.713297 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 08:39:25.713757 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.713739 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 08:39:25.713757 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.713748 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 08:39:25.713853 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.713744 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 08:39:25.713853 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.713741 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\"" Nov 03 08:39:25.714324 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.714308 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 08:39:25.714829 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.714816 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\"" Nov 03 08:39:25.718383 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.718371 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 08:39:25.731536 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.731517 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-678m8"] Nov 03 08:39:25.758118 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.758118 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.758118 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.758323 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdr9\" (UniqueName: \"kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.758323 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.758323 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.758473 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.758473 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxx6m\" (UniqueName: \"kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.758473 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.758586 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.758586 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.758571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2qn\" (UniqueName: \"kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.804026 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.804004 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 08:39:25.859499 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx6m\" (UniqueName: \"kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.859579 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.859579 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.859579 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2qn\" (UniqueName: \"kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.859757 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.859836 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859771 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.859836 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.859836 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.859813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdr9\" (UniqueName: \"kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.859849 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.859910 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.359893717 +0000 UTC m=+31.511565131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : configmap "kube-rbac-proxy" not found Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860082 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860119 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.860131 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.860132 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:25.860184 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.860519 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.860197 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.360181489 +0000 UTC m=+31.511852895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:25.860519 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:25.860226 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls podName:7b492e70-992c-4975-9153-dbc89048db4b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.360209573 +0000 UTC m=+31.511880969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls") pod "dns-default-45lmv" (UID: "7b492e70-992c-4975-9153-dbc89048db4b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:25.860519 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860320 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.860519 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.860408 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.863620 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.863596 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.925286 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.925265 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdr9\" (UniqueName: \"kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:25.925398 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.925284 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx6m\" (UniqueName: \"kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:25.925452 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.925393 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2qn\" (UniqueName: \"kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:25.976400 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:25.976379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:26.035997 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.034830 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xsqnh"] Nov 03 08:39:26.037274 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.037249 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.097547 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097522 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xsqnh"] Nov 03 08:39:26.097751 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097732 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 08:39:26.097804 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097744 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 08:39:26.097804 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097758 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 08:39:26.097889 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097831 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgq6p\"" Nov 03 08:39:26.097889 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.097877 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 08:39:26.121983 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.121960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:26.162870 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.162848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rx4w\" (UniqueName: \"kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.162983 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.162965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.163022 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.162996 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.256636 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.256615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:26.263785 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.263763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.263891 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.263795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.263891 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.263838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx4w\" (UniqueName: \"kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.264429 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.264410 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.266283 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.266267 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.299309 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.299272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:26.299309 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.299279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:26.299406 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.299272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:26.323281 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.323264 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx4w\" (UniqueName: \"kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.350100 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.350077 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:26.364941 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.364916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:26.365054 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.364962 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:26.365054 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.364981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:26.365054 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365025 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:26.365231 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365101 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.365082543 +0000 UTC m=+32.516753949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : configmap "kube-rbac-proxy" not found Nov 03 08:39:26.365231 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365106 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:26.365231 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365170 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls podName:7b492e70-992c-4975-9153-dbc89048db4b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.365156427 +0000 UTC m=+32.516827836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls") pod "dns-default-45lmv" (UID: "7b492e70-992c-4975-9153-dbc89048db4b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:26.365374 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365233 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:26.365374 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:26.365263 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.365253945 +0000 UTC m=+32.516925336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:26.365935 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.365916 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-678m8"] Nov 03 08:39:26.368980 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:39:26.368958 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae96a70_69ca_4de6_a9f5_8b7202bffba8.slice/crio-994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5 WatchSource:0}: Error finding container 994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5: Status 404 returned error can't find the container with id 994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5 Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.401983 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402120 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402132 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402255 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402323 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402408 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 08:39:26.402827 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.402435 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 08:39:26.510722 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.510701 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-678m8" event={"ID":"0ae96a70-69ca-4de6-a9f5-8b7202bffba8","Type":"ContainerStarted","Data":"994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5"} Nov 03 08:39:26.712971 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:26.712854 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xsqnh"] Nov 03 08:39:26.715677 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:39:26.715654 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bbce16_077e_4e80_8085_26d639826e05.slice/crio-1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6 WatchSource:0}: Error finding container 1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6: Status 404 returned error can't find the container with id 1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6 Nov 03 08:39:27.372585 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:27.372551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:27.372606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:27.372634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372676 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372731 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372753 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:27.372781 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372737 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.37271843 +0000 UTC m=+34.524389839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : configmap "kube-rbac-proxy" not found Nov 03 08:39:27.373029 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372807 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.372788344 +0000 UTC m=+34.524459741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:27.373029 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:27.372834 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls podName:7b492e70-992c-4975-9153-dbc89048db4b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.372822871 +0000 UTC m=+34.524494267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls") pod "dns-default-45lmv" (UID: "7b492e70-992c-4975-9153-dbc89048db4b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:27.515433 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:27.515387 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" event={"ID":"42bbce16-077e-4e80-8085-26d639826e05","Type":"ContainerStarted","Data":"1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6"} Nov 03 08:39:27.588536 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:27.588498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:28.029397 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:28.029359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:28.175562 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:28.175497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:28.639751 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:28.639722 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:29.087277 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.087186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:29.087445 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.087363 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 08:39:29.087445 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.087435 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs podName:bec20a64-e1e9-49fc-bb02-d62c451121e1 nodeName:}" failed. No retries permitted until 2026-11-03 08:40:01.087415129 +0000 UTC m=+66.239086534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs") pod "network-metrics-daemon-d8jcr" (UID: "bec20a64-e1e9-49fc-bb02-d62c451121e1") : secret "metrics-daemon-secret" not found Nov 03 08:39:29.188256 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.187790 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:29.191129 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.191086 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:29.287455 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.287354 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pr47_2f00260d-22ef-44e7-802c-d1b9b9585b61/dns-node-resolver/0.log" Nov 03 08:39:29.288711 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.288677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:29.291859 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.291839 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:29.315750 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.315728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:29.320292 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.320106 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:29.389664 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.389637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:29.389783 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.389674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:29.389783 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.389691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:29.389890 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.389798 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 08:39:29.389890 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.389842 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls podName:7b492e70-992c-4975-9153-dbc89048db4b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:33.389830264 +0000 UTC m=+38.541501666 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls") pod "dns-default-45lmv" (UID: "7b492e70-992c-4975-9153-dbc89048db4b") : secret "dns-default-metrics-tls" not found Nov 03 08:39:29.390259 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.390163 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 08:39:29.390259 ip-10-0-131-117 kubenswrapper[2562]: E1103 08:39:29.390234 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls podName:1c816c18-0b4b-42a4-95cd-dde80e6e75bd nodeName:}" failed. No retries permitted until 2026-11-03 08:39:33.390216887 +0000 UTC m=+38.541888291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pd45j" (UID: "1c816c18-0b4b-42a4-95cd-dde80e6e75bd") : secret "insights-runtime-extractor-tls" not found Nov 03 08:39:29.390487 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.390467 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:29.523128 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.523101 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-678m8" event={"ID":"0ae96a70-69ca-4de6-a9f5-8b7202bffba8","Type":"ContainerStarted","Data":"d5d03f886696b389a3bd387aaf0f7b7dc8503f483f7890ef6eeab893a0d945d4"} Nov 03 08:39:29.642211 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.642154 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-678m8" podStartSLOduration=1.617609734 podStartE2EDuration="4.642122795s" totalImagesPullingTime="3.024513061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:29.641325543 +0000 UTC m=+34.792996953" watchObservedRunningTime="2026-11-03 08:39:29.642122795 +0000 UTC m=+34.793794244" Nov 03 08:39:29.751933 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.751785 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkzjz"] Nov 03 08:39:29.755471 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:39:29.755438 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61e725e_6128_4977_9ac4_ee0df1feb8db.slice/crio-ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f WatchSource:0}: Error finding container ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f: Status 404 returned error can't find the container with id ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f Nov 03 08:39:29.802343 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:29.802317 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w4gf5"] Nov 03 08:39:29.804743 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:39:29.804721 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d926adb_98d1_46bb_a1e9_8624624c0f2f.slice/crio-6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6 WatchSource:0}: Error finding container 6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6: Status 404 returned error can't find the container with id 6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6 Nov 03 08:39:30.283705 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.283675 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dfmxv_a57c4f2b-51b2-43d1-9fa4-052405923764/node-ca/0.log" Nov 03 08:39:30.526767 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.526734 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" event={"ID":"42bbce16-077e-4e80-8085-26d639826e05","Type":"ContainerStarted","Data":"2f136c4d531f579a1a60b9ec75b6d06d9765ec5ab461bfc39063f8e7f1d17ec7"} Nov 03 08:39:30.528015 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.527989 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w4gf5" event={"ID":"7d926adb-98d1-46bb-a1e9-8624624c0f2f","Type":"ContainerStarted","Data":"6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6"} Nov 03 08:39:30.529197 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.529169 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qkzjz" event={"ID":"f61e725e-6128-4977-9ac4-ee0df1feb8db","Type":"ContainerStarted","Data":"ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f"} Nov 03 08:39:30.642128 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.642084 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" podStartSLOduration=2.853392521 podStartE2EDuration="5.642072236s" totalImagesPullingTime="2.788679715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:30.641600369 +0000 UTC m=+35.793271779" watchObservedRunningTime="2026-11-03 08:39:30.642072236 +0000 UTC m=+35.793743647" Nov 03 08:39:30.887115 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.887081 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-678m8_0ae96a70-69ca-4de6-a9f5-8b7202bffba8/serve-healthcheck-canary/0.log" Nov 03 08:39:30.910021 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:30.909986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:31.223609 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:31.223506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:31.370729 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:31.370659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:31.663892 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:31.663852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:31.802498 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:31.802472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:32.689164 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:32.689078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:32.818642 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:32.818605 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:32.996916 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:32.996816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:33.167788 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.167752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:33.366166 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.366120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:33.442228 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.442184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:33.442415 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.442236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:33.445043 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.445016 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:33.445172 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.445019 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:33.459223 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.459008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:33.474321 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.474298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:33.574401 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.574366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:33.700773 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.700665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:33.922459 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.922429 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45lmv"] Nov 03 08:39:33.963255 ip-10-0-131-117 kubenswrapper[2562]: W1103 08:39:33.963224 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b492e70_992c_4975_9153_dbc89048db4b.slice/crio-01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e WatchSource:0}: Error finding container 01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e: Status 404 returned error can't find the container with id 01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e Nov 03 08:39:33.977796 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:33.977770 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pd45j"] Nov 03 08:39:34.543820 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.543744 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd45j" event={"ID":"1c816c18-0b4b-42a4-95cd-dde80e6e75bd","Type":"ContainerStarted","Data":"6dba99a3aee9d955339df4eeea502b5032622646afaeb230976c07cdff30aa92"} Nov 03 08:39:34.543820 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.543807 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd45j" event={"ID":"1c816c18-0b4b-42a4-95cd-dde80e6e75bd","Type":"ContainerStarted","Data":"b5154674ec06427473b74e3e3eb8136ba5aa843513cfb77bbebccd391ab7d77c"} Nov 03 08:39:34.545221 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.545181 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lmv" event={"ID":"7b492e70-992c-4975-9153-dbc89048db4b","Type":"ContainerStarted","Data":"01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e"} Nov 03 08:39:34.547007 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.546958 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w4gf5" event={"ID":"7d926adb-98d1-46bb-a1e9-8624624c0f2f","Type":"ContainerStarted","Data":"e093a0187923bf3f62186fb01bbaf96b02b6e9ea07143bd9ace20a382401859f"} Nov 03 08:39:34.547170 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.547131 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:34.572375 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.572345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:34.664168 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:34.663275 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w4gf5" podStartSLOduration=34.275399457 podStartE2EDuration="38.663257958s" totalImagesPullingTime="4.387858501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:34.663035172 +0000 UTC m=+39.814706586" watchObservedRunningTime="2026-11-03 08:39:34.663257958 +0000 UTC m=+39.814929349" Nov 03 08:39:35.012881 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:35.012822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:35.169284 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:35.169230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:35.366639 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:35.366608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:35.769451 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:35.769380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:36.001340 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:36.001312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:36.134798 ip-10-0-131-117 kubenswrapper[2562]: I1103 08:39:36.134763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:36.221464 ip-10-0-131-117 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 08:39:36.235765 ip-10-0-131-117 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 08:39:36.235932 ip-10-0-131-117 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 08:39:36.236000 ip-10-0-131-117 systemd[1]: kubelet.service: Consumed 2.420s CPU time. Nov 03 08:39:36.248836 ip-10-0-131-117 systemd[1]: Starting Kubernetes Kubelet... Nov 03 08:39:36.360296 ip-10-0-131-117 kubenswrapper[4543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:36.360296 ip-10-0-131-117 kubenswrapper[4543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 08:39:36.360296 ip-10-0-131-117 kubenswrapper[4543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:36.360296 ip-10-0-131-117 kubenswrapper[4543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 08:39:36.362345 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362276 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.362345 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362343 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.362345 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362348 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362351 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362355 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362358 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362361 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362363 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362366 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362369 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362371 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362374 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362376 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362379 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362382 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362384 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362387 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362389 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362392 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362394 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362397 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.362455 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362399 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362402 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362404 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362407 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362410 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362418 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362421 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362423 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362426 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362428 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362430 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362435 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362439 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362442 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362446 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362451 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362453 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362456 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362459 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362462 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.362904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362464 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362468 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362471 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362474 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362476 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362479 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362481 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362484 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362486 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362489 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362491 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362494 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362496 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362499 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362502 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362505 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362507 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362510 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362512 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362515 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.363391 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362518 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362520 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362523 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362525 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362528 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362530 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362533 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362535 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362538 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362540 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362542 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362545 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362547 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362550 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362552 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362555 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362557 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362560 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362562 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362565 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.363863 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362567 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362569 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362572 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362574 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362577 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362579 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362582 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362584 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362587 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362589 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362592 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362595 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362597 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362599 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362602 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362604 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362607 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362610 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362613 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.364354 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362615 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362618 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362620 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362623 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362625 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.362627 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363396 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363404 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363407 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363410 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363413 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363416 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363418 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363421 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363424 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363428 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363432 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363435 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363437 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.364810 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363440 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363443 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363446 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363448 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363451 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363453 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363456 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363458 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363461 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363463 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363466 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363468 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363471 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363473 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363476 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363478 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363480 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363483 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363485 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363488 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.365266 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363491 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363493 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363496 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363498 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363501 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363503 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363506 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363508 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363511 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363513 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363516 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363518 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363520 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363523 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363525 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363528 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363530 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363533 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363536 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.365773 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363538 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363541 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363543 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363545 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363548 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363550 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363553 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363556 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363560 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363563 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363566 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363569 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363572 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363574 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363577 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363580 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363582 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363585 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363587 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.366258 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363590 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363592 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363595 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363597 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363599 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363602 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363604 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363607 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363609 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363612 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363615 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363618 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363620 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363623 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363625 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363628 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363630 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363633 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363635 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363638 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.366715 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363641 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363644 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363646 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363649 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363651 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363653 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363656 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363658 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363661 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363663 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363666 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363668 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363670 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363673 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.363675 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364182 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364189 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364192 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364195 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364198 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.367207 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364201 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364203 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364212 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364215 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364218 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364221 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364223 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364226 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364228 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364231 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364233 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364236 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364238 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364241 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364243 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364245 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364248 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364250 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364253 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364256 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.367689 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364258 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364261 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364263 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364265 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364268 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364271 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364273 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364276 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364278 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364281 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364283 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364286 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364289 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364291 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364294 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364302 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364305 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364308 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364310 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364313 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.368223 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364315 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364318 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364320 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364322 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364325 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364327 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364330 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364334 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364342 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364345 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364348 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364353 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364356 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364358 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364361 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364364 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364366 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364369 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364371 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364374 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.368705 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364376 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364379 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364383 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364386 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364389 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364392 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364394 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364397 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364405 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364410 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364412 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364415 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364417 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364420 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364422 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364425 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364427 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364430 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364433 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.369227 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364435 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364437 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364440 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364442 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364446 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364448 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364451 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364454 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364456 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364458 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364461 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364463 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364466 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364468 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364471 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364473 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364476 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364478 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364481 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.369787 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364485 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364487 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.364489 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364603 4543 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364614 4543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364627 4543 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364632 4543 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364637 4543 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364640 4543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364644 4543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364648 4543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364651 4543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364654 4543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364657 4543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364660 4543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364664 4543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364666 4543 flags.go:64] FLAG: --cgroup-root="" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364671 4543 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364674 4543 flags.go:64] FLAG: --client-ca-file="" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364677 4543 flags.go:64] FLAG: --cloud-provider="external" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364680 4543 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364686 4543 flags.go:64] FLAG: --cluster-domain="" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364689 4543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 08:39:36.370284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364691 4543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364694 4543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364697 4543 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364701 4543 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364704 4543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364707 4543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364710 4543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364713 4543 flags.go:64] FLAG: --contention-profiling="false" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364716 4543 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364719 4543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364722 4543 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364725 4543 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364730 4543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364739 4543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364742 4543 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364745 4543 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364748 4543 flags.go:64] FLAG: --enable-server="true" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364750 4543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364757 4543 flags.go:64] FLAG: --event-burst="100" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364760 4543 flags.go:64] FLAG: --event-qps="50" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364763 4543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364766 4543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364768 4543 flags.go:64] FLAG: --eviction-hard="" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364773 4543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 08:39:36.370817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364775 4543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364778 4543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364783 4543 flags.go:64] FLAG: --eviction-soft="" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364786 4543 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364788 4543 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364791 4543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364794 4543 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364797 4543 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364799 4543 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364802 4543 flags.go:64] FLAG: --feature-gates="" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364806 4543 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364809 4543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364812 4543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364816 4543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364819 4543 flags.go:64] FLAG: --healthz-port="10248" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364822 4543 flags.go:64] FLAG: --help="false" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364825 4543 flags.go:64] FLAG: --hostname-override="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364828 4543 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364831 4543 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364833 4543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364838 4543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364841 4543 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364845 4543 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 08:39:36.371446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364848 4543 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364851 4543 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364854 4543 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364856 4543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364859 4543 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364862 4543 flags.go:64] FLAG: --kube-reserved="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364865 4543 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364867 4543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364870 4543 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364873 4543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364876 4543 flags.go:64] FLAG: --lock-file="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364880 4543 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364883 4543 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364886 4543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364892 4543 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364894 4543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364897 4543 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364900 4543 flags.go:64] FLAG: --logging-format="text" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364903 4543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364906 4543 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364909 4543 flags.go:64] FLAG: --manifest-url="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364911 4543 flags.go:64] FLAG: --manifest-url-header="" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364918 4543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364921 4543 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364925 4543 flags.go:64] FLAG: --max-pods="110" Nov 03 08:39:36.372415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364928 4543 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364930 4543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364933 4543 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364936 4543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364940 4543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364943 4543 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364946 4543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364953 4543 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364956 4543 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364959 4543 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364962 4543 flags.go:64] FLAG: --pod-cidr="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364965 4543 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364967 4543 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364970 4543 flags.go:64] FLAG: --pods-per-core="0" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364973 4543 flags.go:64] FLAG: --port="10250" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364976 4543 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364979 4543 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0a2369d01d0fa5665" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364982 4543 flags.go:64] FLAG: --qos-reserved="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364986 4543 flags.go:64] FLAG: --read-only-port="10255" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364989 4543 flags.go:64] FLAG: --register-node="true" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364992 4543 flags.go:64] FLAG: --register-with-taints="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364996 4543 flags.go:64] FLAG: --registry-burst="10" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.364998 4543 flags.go:64] FLAG: --registry-qps="5" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365001 4543 flags.go:64] FLAG: --reserved-cpus="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365004 4543 flags.go:64] FLAG: --reserved-memory="" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365008 4543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 08:39:36.373050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365011 4543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365013 4543 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365016 4543 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365019 4543 flags.go:64] FLAG: --runonce="false" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365023 4543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365025 4543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365028 4543 flags.go:64] FLAG: --seccomp-default="false" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365031 4543 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365034 4543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365037 4543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365041 4543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365044 4543 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365046 4543 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365049 4543 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365052 4543 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365055 4543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365058 4543 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365061 4543 flags.go:64] FLAG: --system-cgroups="" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365064 4543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365069 4543 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365071 4543 flags.go:64] FLAG: --tls-cert-file="" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365074 4543 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365081 4543 flags.go:64] FLAG: --tls-min-version="" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365083 4543 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365087 4543 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 08:39:36.373715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365090 4543 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365093 4543 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365095 4543 flags.go:64] FLAG: --v="2" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365099 4543 flags.go:64] FLAG: --version="false" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365103 4543 flags.go:64] FLAG: --vmodule="" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365107 4543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365110 4543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365291 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365296 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365299 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365302 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365304 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365306 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365309 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365311 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365314 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365317 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365321 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365323 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365326 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365328 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.374325 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365331 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365334 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365336 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365339 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365341 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365344 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365346 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365349 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365351 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365354 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365359 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365361 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365365 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365369 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365373 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365376 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365379 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365381 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365384 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.374849 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365387 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365389 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365392 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365394 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365397 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365399 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365402 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365405 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365407 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365410 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365413 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365415 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365417 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365420 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365422 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365425 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365428 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365431 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365433 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365435 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.375362 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365438 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365441 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365443 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365447 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365449 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365452 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365454 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365457 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365461 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365465 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365468 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365471 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365474 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365477 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365480 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365482 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365485 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365487 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365489 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365492 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.375973 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365494 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365497 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365499 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365502 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365504 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365507 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365509 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365512 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365515 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365518 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365520 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365523 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365525 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365527 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365530 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365534 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365536 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365539 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365541 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365544 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.376619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365547 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365550 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365552 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365554 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365557 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365559 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365562 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365564 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365566 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365569 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365571 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365573 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365576 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.365605 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365636 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365640 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.377209 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365643 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365646 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365649 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365651 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365654 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365656 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365659 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365663 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365666 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365670 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365672 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365675 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365678 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365681 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365684 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365687 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365690 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365693 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365695 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.377651 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365698 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365700 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365703 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365705 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365708 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365711 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365713 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365716 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365718 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365721 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365723 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365726 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365728 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365731 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365733 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365736 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365738 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365741 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365743 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.378213 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365746 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365748 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365751 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365753 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365757 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365760 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365764 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365766 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365769 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365772 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365775 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365778 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365780 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365783 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365786 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365788 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365790 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365793 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365795 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365798 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.378700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365800 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365802 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365805 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365807 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365811 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365814 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365816 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365818 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365821 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365823 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365826 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365828 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365831 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365833 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365836 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365838 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365841 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365844 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365847 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365849 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.379187 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365851 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365855 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365857 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365860 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365862 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365865 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365867 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365869 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365872 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365874 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365876 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365879 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365881 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365883 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365886 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365888 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365891 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365893 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365896 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.379680 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365899 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365901 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365904 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365906 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365909 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365911 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.365914 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.369536 4543 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.369551 4543 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.369568 4543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.369572 4543 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369624 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369628 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369632 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369637 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369641 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369644 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369647 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369650 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369652 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.380124 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369655 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369658 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369661 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369664 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369666 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369669 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369672 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369674 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369677 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369679 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369682 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369685 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369688 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369690 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369694 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369699 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369702 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369705 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369709 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369712 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.380628 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369714 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369716 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369719 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369721 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369724 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369726 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369728 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369731 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369733 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369735 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369738 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369740 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369743 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369745 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369748 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369750 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369752 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369755 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369758 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369760 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.381105 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369763 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369765 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369767 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369771 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369773 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369776 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369779 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369781 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369784 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369786 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369788 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369791 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369794 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369796 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369799 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369802 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369804 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369806 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369809 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.381617 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369811 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369814 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369816 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369818 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369821 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369823 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369825 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369828 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369830 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369833 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369835 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369838 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369841 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369843 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369846 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369849 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369851 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369854 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369856 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.382073 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369858 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369861 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369863 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369866 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369868 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369870 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369872 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369875 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369877 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369879 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369882 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369885 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369887 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369889 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369892 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369894 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369897 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369899 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.382667 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.369902 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.369928 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370069 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370074 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370077 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370080 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370082 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370085 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370088 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370090 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370093 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370095 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370098 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370100 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370102 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370106 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.383107 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370110 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370113 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370116 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370119 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370121 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370152 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370156 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370159 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370161 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370163 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370166 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370168 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370171 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370173 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370176 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370178 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370181 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370184 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370186 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370189 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.383505 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370191 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370194 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370196 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370199 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370202 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370205 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370207 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370210 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370212 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370214 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370217 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370219 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370221 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370224 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370227 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370229 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370231 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370234 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370236 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.383985 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370239 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370241 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370243 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370246 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370248 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370253 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370255 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370258 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370260 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370263 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370265 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370267 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370270 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370272 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370275 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370277 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370280 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370282 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370285 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.384483 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370288 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370292 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370295 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370298 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370301 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370304 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370306 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370309 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370311 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370313 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370316 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370318 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370321 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370323 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370325 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370328 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370330 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370333 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370337 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.384950 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370339 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370342 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370344 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370347 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370350 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370352 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370355 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370357 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370360 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370362 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370364 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370367 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370369 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370373 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370376 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.370399 4543 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 08:39:36.385431 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370429 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370434 4543 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370437 4543 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370440 4543 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370442 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370445 4543 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370447 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370450 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370452 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370454 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370458 4543 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370460 4543 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370464 4543 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370467 4543 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370469 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370472 4543 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370476 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370478 4543 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370481 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 08:39:36.385864 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370483 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370486 4543 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370488 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370491 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370493 4543 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370496 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370498 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370501 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370503 4543 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370505 4543 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370508 4543 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370511 4543 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370514 4543 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370516 4543 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370519 4543 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370521 4543 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370524 4543 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370526 4543 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370529 4543 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370531 4543 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 08:39:36.386365 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370533 4543 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370536 4543 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370538 4543 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370541 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370543 4543 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370547 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370550 4543 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370552 4543 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370555 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370559 4543 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370561 4543 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370563 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370566 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370568 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370571 4543 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370575 4543 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370578 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370581 4543 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370584 4543 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 08:39:36.387004 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370586 4543 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370589 4543 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370591 4543 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370593 4543 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370596 4543 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370599 4543 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370602 4543 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370604 4543 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370607 4543 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370609 4543 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370612 4543 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370614 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370617 4543 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370620 4543 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370622 4543 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370624 4543 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370627 4543 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370629 4543 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370633 4543 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 08:39:36.387784 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370635 4543 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370638 4543 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370640 4543 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370644 4543 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370646 4543 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370649 4543 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370651 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370654 4543 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370656 4543 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370659 4543 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370662 4543 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370664 4543 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370667 4543 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370669 4543 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370671 4543 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370674 4543 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370677 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370679 4543 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370682 4543 feature_gate.go:386] unrecognized feature gate: Example Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370685 4543 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 08:39:36.388660 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370687 4543 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370689 4543 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370692 4543 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370696 4543 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370699 4543 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370702 4543 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370704 4543 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370707 4543 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:36.370709 4543 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.370756 4543 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.372795 4543 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.372858 4543 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.373058 4543 server.go:1014] "Starting client certificate rotation" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.373169 4543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.373212 4543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:56 +0000 UTC" deadline="2028-04-26 20:38:35.08296562 +0000 UTC" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.373245 4543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12971h58m58.709722086s" Nov 03 08:39:36.389252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.373564 4543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:39:36.389637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.374567 4543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 08:39:36.389637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.375919 4543 log.go:25] "Validated CRI v1 runtime API" Nov 03 08:39:36.389637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.377468 4543 log.go:25] "Validated CRI v1 image API" Nov 03 08:39:36.389637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.378196 4543 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 08:39:36.389637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.379623 4543 fs.go:135] Filesystem UUIDs: map[5156017e-e355-44d5-adcc-4ce7e35c91c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0361df5-5b88-4966-948a-a62158183dd8:/dev/nvme0n1p3] Nov 03 08:39:36.389770 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.379641 4543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5154674ec06427473b74e3e3eb8136ba5aa843513cfb77bbebccd391ab7d77c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5154674ec06427473b74e3e3eb8136ba5aa843513cfb77bbebccd391ab7d77c/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~projected/kube-api-access-6r2qn:{mountpoint:/var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~projected/kube-api-access-6r2qn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15fbcb5f-5b60-488c-929a-e5661647fb3a/volumes/kubernetes.io~projected/kube-api-access-z86ww:{mountpoint:/var/lib/kubelet/pods/15fbcb5f-5b60-488c-929a-e5661647fb3a/volumes/kubernetes.io~projected/kube-api-access-z86ww major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~projected/kube-api-access-fxx6m:{mountpoint:/var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~projected/kube-api-access-fxx6m major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~projected/kube-api-access-qk2f5:{mountpoint:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~projected/kube-api-access-qk2f5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f00260d-22ef-44e7-802c-d1b9b9585b61/volumes/kubernetes.io~projected/kube-api-access-c752x:{mountpoint:/var/lib/kubelet/pods/2f00260d-22ef-44e7-802c-d1b9b9585b61/volumes/kubernetes.io~projected/kube-api-access-c752x major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~projected/kube-api-access-8rx4w:{mountpoint:/var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~projected/kube-api-access-8rx4w major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~secret/signing-key major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/518c5a4f-9809-452d-8a8d-aea90d5961c5/volumes/kubernetes.io~projected/kube-api-access-kjvxg:{mountpoint:/var/lib/kubelet/pods/518c5a4f-9809-452d-8a8d-aea90d5961c5/volumes/kubernetes.io~projected/kube-api-access-kjvxg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/530f4286-b8c2-4331-9bcc-372807e290d9/volumes/kubernetes.io~projected/kube-api-access-2hl8n:{mountpoint:/var/lib/kubelet/pods/530f4286-b8c2-4331-9bcc-372807e290d9/volumes/kubernetes.io~projected/kube-api-access-2hl8n major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~projected/kube-api-access-bkdr9:{mountpoint:/var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~projected/kube-api-access-bkdr9 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d926adb-98d1-46bb-a1e9-8624624c0f2f/volumes/kubernetes.io~projected/kube-api-access-jkkn2:{mountpoint:/var/lib/kubelet/pods/7d926adb-98d1-46bb-a1e9-8624624c0f2f/volumes/kubernetes.io~projected/kube-api-access-jkkn2 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c823c7a-c6fb-4796-9a85-5f23e6c05759/volumes/kubernetes.io~projected/kube-api-access-rtnlg:{mountpoint:/var/lib/kubelet/pods/8c823c7a-c6fb-4796-9a85-5f23e6c05759/volumes/kubernetes.io~projected/kube-api-access-rtnlg major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a57c4f2b-51b2-43d1-9fa4-052405923764/volumes/kubernetes.io~projected/kube-api-access-dctsb:{mountpoint:/var/lib/kubelet/pods/a57c4f2b-51b2-43d1-9fa4-052405923764/volumes/kubernetes.io~projected/kube-api-access-dctsb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~projected/kube-api-access-5npcm:{mountpoint:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~projected/kube-api-access-5npcm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bec20a64-e1e9-49fc-bb02-d62c451121e1/volumes/kubernetes.io~projected/kube-api-access-gncrg:{mountpoint:/var/lib/kubelet/pods/bec20a64-e1e9-49fc-bb02-d62c451121e1/volumes/kubernetes.io~projected/kube-api-access-gncrg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0527392-1028-47e0-a542-80038e79f6da/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e0527392-1028-47e0-a542-80038e79f6da/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f61e725e-6128-4977-9ac4-ee0df1feb8db/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f61e725e-6128-4977-9ac4-ee0df1feb8db/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/adf0201b60a501c4960719e12c890d35921a965b81b0b092575c9aa5f1f7ffe4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b959b441ddd9553839498378868c41ff6460274a28d08049fcfbeb5311e3b2b9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e59bcac11b236b121f04ebfefe704fc51e42032a6e5f4a485ea4a374d90a4f90/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/aa1341be4df55bc8721b6cdc6ac0ef567e7425426f24ad00c4b4c88530f5fb6f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/505c3a07ae665a4b5cc755e56475030478daa504df42b0fdb0cef5ca280b8450/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bb733e2ed93b18a1ff1722c6ce84ee2278c3e08b7e907cac5f0314730303eab6/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/044a2f05da81f84a772488e51c981e58fda37a066421bfa04fe7ca3d23bcaed1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4ea77e8e3f074e7eb76f490ad55c67074ea95540231503671896fd4b83d3dab0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/12c04e8cdfa2ab62d91f79aec74e32640ae4dce6d1a3f76a027d649bae5ed9b7/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/826370eecceef16c0f350d702c5f421b3f19fca2ce332df03d59eac3e5044b95/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f6b2e925d712aa45c4ee603cfbcd0c9b0bbbbca7c79b66eeb60a9f21e7ed0239/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/6b66b47d540444fe598b1f61966e3f90e4963bdaebee3dc5e5fd8de3e163e9b5/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7b4f5b6399e717f037dfe07fae2b4bcc983a3496e8e8d5402f371f1b29b7326f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f1041ceb3cd7fb621dee05edabf18c7fc5edfd6589bdaa01709efb24bc9f1a02/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/89140523bd587e641568035733c50ef49401cb8c600212fc769bf789d9bd1877/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5908517c81e72b2e3303c3295074c33583850d4983cb7a49f770cb0e8d53d853/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/106ce76053b668720549cca804edcfc8b2609c291e4459482688872f5069e482/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/782ffa2dea0d4aef8a676145e7786bf7e78c9cc0c3b1a01a0b27372407c6e863/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f7539daea1a67a7cdf58e1256d1d7c551e132e703c31ae4fb85a1bb69107f891/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ead1f3125b59fcc7f369798b4abec5890b7778ab4ca7f42147843dc831d56831/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3439e5e22bc806b96467ae4ab5f227a709a06a23d17dc13c176b5a39b4614953/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/175de3eb9625f7199a198c32457bf195d5c62e351ac7932a213f845509bfa8b7/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/1c0babbfafcbc29a1709823861d17d58ee522dfdcde59c27358efb174ec092be/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3ad444f15096818e56b443f599c3ba2c6125281a74a9f6484940993da976394d/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/abc78c84dd21a6c0cb2a70cee9610caca6ffaeb0bdddc1d9fd2387834a7c608b/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/79e5bbb51e1fb5fca117c874f6d24ed2ede66d81c802d14b70f402fbcee58441/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/7d56d2b59b01fa3d261893977893ecfcde3569aaa505a59375bcca8f83b29c09/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d1028ba2678ce20fdc8cf4f5648c5066d300569bc2112b6e2e68c3f8fda8e332/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/6596dece3ed09e1b10ec89f06a35eef5d56ca24a6c56fd98b8629e1b144e44bf/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/8c324ccb4c3652ec61e4753eb7e3a4ed027eedc9f10d85864adebe60f095c67c/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/650d4efcbb8e8c7475fa3690fb90a09408371d74a746d08bdbc60be7a5e38f52/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/00e50f93dcac535ae3f24fe0d70c1816fe5f513319efd49d0c80746e08316f1c/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ac08a6ee2cd56659ef28718ea9ea830851e56fb7be1f4ce574b93ab56d98985b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5abeba44bcc2138cb720a102a95da0019993f499be85638d6ca3b7c5e5053650/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3a767b5d935882e9111f19fe95eecd583300027c3a67f0621cbcec7df9f2f1b0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e3eaad137dbea767ed930c58b74f26ac86e80e87d98e7304b0c554f685389a1d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1481f066180a22498c256fb3f4df65c39aaf48d9c060814f83d155638df2d8b6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/6513a6bb54c3f538f4a57452bbc1b37b1f12fd59098618c6cebfdb567f436888/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/642c42cecfa9de0fa85567dc574fc27c9e2b75327a42ffc8818774d83c4975f9/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6b31de54228ce8e73f5739e65c2bafe26089fa56880401b0e4951bbf247c7cfa/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/9674e1890a55bf3af07cc2e94c8ae674bdbf34e15ce267f0b9c58327485b35e9/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7c4669d52f9cdaffe3c5ef3070a90910b4e22e891d74437a1b8827f41b357073/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5ed82d0f27afb1babc1d9738adbe292e82ad320903ea6f223228e50063ba4cb4/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.385513 4543 manager.go:217] Machine: {Timestamp:2026-11-03 08:39:36.385055969 +0000 UTC m=+0.053096700 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103459 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2170a9ccd7fad3d313fa31d1cc664d SystemUUID:ec2170a9-ccd7-fad3-d313-fa31d1cc664d BootID:c7773d95-afce-483e-935f-bbf241c6f5bc Filesystems:[{Device:/var/lib/kubelet/pods/bec20a64-e1e9-49fc-bb02-d62c451121e1/volumes/kubernetes.io~projected/kube-api-access-gncrg DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~projected/kube-api-access-5npcm DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1347cf210de489a392efc887f917fbf9c227275680e2567ae0b1af6378f856c6/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d926adb-98d1-46bb-a1e9-8624624c0f2f/volumes/kubernetes.io~projected/kube-api-access-jkkn2 DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/b8977ebc-ddf6-41df-9faf-175c92054114/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~projected/kube-api-access-6r2qn DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~projected/kube-api-access-qk2f5 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94c53e2b7f645bb9331a6bc5d5f8a0012431130662e436251f9cf3102bd81d59/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/994d7c8eb46d55275add72e1e4a38b236808b3ef772b2b684bd69b0c8bbbd2e5/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01e94f36cceeb5c483323036856fcf1ceac41839fa3abd5069005dc8c985021e/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f683c01e9ddd3aae2d9b4d96370c6a5175ec2e295fe33bc23147f2a5a2ab60d3/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f61e725e-6128-4977-9ac4-ee0df1feb8db/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5154674ec06427473b74e3e3eb8136ba5aa843513cfb77bbebccd391ab7d77c/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/284c991793cf09cd4cb4ee73f1c012d7552e663b5dc3e60583920df34e8341d8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/518c5a4f-9809-452d-8a8d-aea90d5961c5/volumes/kubernetes.io~projected/kube-api-access-kjvxg DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e0527392-1028-47e0-a542-80038e79f6da/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e0ea12759642351de6fc75577101da3c4bd1f08d5224660b40f2a9704074dd1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b492e70-992c-4975-9153-dbc89048db4b/volumes/kubernetes.io~projected/kube-api-access-bkdr9 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8c823c7a-c6fb-4796-9a85-5f23e6c05759/volumes/kubernetes.io~projected/kube-api-access-rtnlg DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~projected/kube-api-access-8rx4w DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a57c4f2b-51b2-43d1-9fa4-052405923764/volumes/kubernetes.io~projected/kube-api-access-dctsb DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/42bbce16-077e-4e80-8085-26d639826e05/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15fbcb5f-5b60-488c-929a-e5661647fb3a/volumes/kubernetes.io~projected/kube-api-access-z86ww DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebe2dc177146d5df1ef37b56cb58d2a3d25a1bfe2a3586636502bd7568ce547f/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ae96a70-69ca-4de6-a9f5-8b7202bffba8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f00260d-22ef-44e7-802c-d1b9b9585b61/volumes/kubernetes.io~projected/kube-api-access-c752x DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eb371a711766e45269dfda1609a3e41f21b802400eac9404e04cbe901700f19/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27e5037f-1bbd-4170-8108-6a06e3f782f2/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1c816c18-0b4b-42a4-95cd-dde80e6e75bd/volumes/kubernetes.io~projected/kube-api-access-fxx6m DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eb458d44600be5c8e6423b89456dd3888a603b73a59fd8e05297a4209a5bca1/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a87e692be78d814c69819a1bde7da5d4c293f4016218c21e6b5a24955eb29c6/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cff5a022f6b56944d21c687db12171579fa1bcc648600b7efed88cd3edd37047/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c8365d6ed5063b68a62d1e9dcea6ce2d37eccecf0673105aa89c63aed6fbf13/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/530f4286-b8c2-4331-9bcc-372807e290d9/volumes/kubernetes.io~projected/kube-api-access-2hl8n DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01e94f36cceeb5c MacAddress:46:aa:f8:4f:cd:47 Speed:10000 Mtu:8901} {Name:1347cf210de489a MacAddress:d6:f4:9a:25:e1:90 Speed:10000 Mtu:8901} {Name:6a87e692be78d81 MacAddress:c6:5f:4a:a9:0c:2c Speed:10000 Mtu:8901} {Name:994d7c8eb46d552 MacAddress:e2:33:96:8c:d9:96 Speed:10000 Mtu:8901} {Name:b5154674ec06427 MacAddress:7e:5c:fa:55:eb:be Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:12:cf:41:b0:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:06:c5:41:50:f4:5f Speed:0 Mtu:8901} {Name:ebe2dc177146d5d MacAddress:f6:a9:34:23:bb:70 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:12:cf:41:b0:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:5a:57:3d:ce:c4:1c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:00:8f:86:cb:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.385763 4543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.385799 4543 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386052 4543 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386075 4543 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-117.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386223 4543 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386231 4543 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386244 4543 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386262 4543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386337 4543 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386481 4543 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386570 4543 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386600 4543 kubelet.go:493] "Attempting to sync node with API server" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386612 4543 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386628 4543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386638 4543 kubelet.go:405] "Adding apiserver pod source" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.386649 4543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.387900 4543 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.387916 4543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:39:36.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.388683 4543 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 08:39:36.398260 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398243 4543 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 08:39:36.398260 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398258 4543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 08:39:36.398392 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398351 4543 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 08:39:36.398542 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398530 4543 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 08:39:36.398573 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398560 4543 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 08:39:36.398676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398667 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398682 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398688 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398693 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398699 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398705 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 08:39:36.398712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398710 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398718 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398727 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398733 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398741 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398749 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398805 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 08:39:36.398894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.398814 4543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 08:39:36.402071 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.402057 4543 server.go:1263] "Started kubelet" Nov 03 08:39:36.402218 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.402163 4543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 08:39:36.403621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.402464 4543 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 08:39:36.403621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.402495 4543 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 08:39:36.403621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.403101 4543 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 08:39:36.402566 ip-10-0-131-117 systemd[1]: Started Kubernetes Kubelet. Nov 03 08:39:36.403984 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.403740 4543 server.go:317] "Adding debug handlers to kubelet server" Nov 03 08:39:36.404972 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.404937 4543 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 08:39:36.404972 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.404968 4543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 08:39:36.405352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.405028 4543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:58 +0000 UTC" deadline="2028-06-09 20:51:07.436180825 +0000 UTC" Nov 03 08:39:36.405352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.405046 4543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14028h11m31.03113756s" Nov 03 08:39:36.405352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.405108 4543 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 08:39:36.405352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.405116 4543 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 08:39:36.405352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.405169 4543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 08:39:36.406362 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.406117 4543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.407950 4543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.407969 4543 factory.go:55] Registering systemd factory Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.407978 4543 factory.go:223] Registration of the systemd container factory successfully Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408290 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8977ebc-ddf6-41df-9faf-175c92054114" volumeName="kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408340 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0527392-1028-47e0-a542-80038e79f6da" volumeName="kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408357 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492e70-992c-4975-9153-dbc89048db4b" volumeName="kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408377 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8977ebc-ddf6-41df-9faf-175c92054114" volumeName="kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408406 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e5037f-1bbd-4170-8108-6a06e3f782f2" volumeName="kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408425 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae96a70-69ca-4de6-a9f5-8b7202bffba8" volumeName="kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408438 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57c4f2b-51b2-43d1-9fa4-052405923764" volumeName="kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408451 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0527392-1028-47e0-a542-80038e79f6da" volumeName="kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408488 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e5037f-1bbd-4170-8108-6a06e3f782f2" volumeName="kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408502 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e5037f-1bbd-4170-8108-6a06e3f782f2" volumeName="kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408520 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="530f4286-b8c2-4331-9bcc-372807e290d9" volumeName="kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408533 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d926adb-98d1-46bb-a1e9-8624624c0f2f" volumeName="kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408569 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c823c7a-c6fb-4796-9a85-5f23e6c05759" volumeName="kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408599 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8977ebc-ddf6-41df-9faf-175c92054114" volumeName="kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408640 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae96a70-69ca-4de6-a9f5-8b7202bffba8" volumeName="kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408656 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c816c18-0b4b-42a4-95cd-dde80e6e75bd" volumeName="kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408668 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c816c18-0b4b-42a4-95cd-dde80e6e75bd" volumeName="kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408686 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492e70-992c-4975-9153-dbc89048db4b" volumeName="kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408714 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bec20a64-e1e9-49fc-bb02-d62c451121e1" volumeName="kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408735 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c816c18-0b4b-42a4-95cd-dde80e6e75bd" volumeName="kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408765 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42bbce16-077e-4e80-8085-26d639826e05" volumeName="kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408800 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="518c5a4f-9809-452d-8a8d-aea90d5961c5" volumeName="kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408815 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492e70-992c-4975-9153-dbc89048db4b" volumeName="kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408827 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e5037f-1bbd-4170-8108-6a06e3f782f2" volumeName="kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408844 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fbcb5f-5b60-488c-929a-e5661647fb3a" volumeName="kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408874 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42bbce16-077e-4e80-8085-26d639826e05" volumeName="kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408892 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="518c5a4f-9809-452d-8a8d-aea90d5961c5" volumeName="kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408904 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b492e70-992c-4975-9153-dbc89048db4b" volumeName="kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408943 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f61e725e-6128-4977-9ac4-ee0df1feb8db" volumeName="kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408959 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fbcb5f-5b60-488c-929a-e5661647fb3a" volumeName="kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408977 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e5037f-1bbd-4170-8108-6a06e3f782f2" volumeName="kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.408992 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f00260d-22ef-44e7-802c-d1b9b9585b61" volumeName="kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409016 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f00260d-22ef-44e7-802c-d1b9b9585b61" volumeName="kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409036 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a57c4f2b-51b2-43d1-9fa4-052405923764" volumeName="kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409048 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="530f4286-b8c2-4331-9bcc-372807e290d9" volumeName="kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409065 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fbcb5f-5b60-488c-929a-e5661647fb3a" volumeName="kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409079 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fbcb5f-5b60-488c-929a-e5661647fb3a" volumeName="kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409095 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c816c18-0b4b-42a4-95cd-dde80e6e75bd" volumeName="kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409112 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42bbce16-077e-4e80-8085-26d639826e05" volumeName="kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409125 4543 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="518c5a4f-9809-452d-8a8d-aea90d5961c5" volumeName="kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg" seLinuxMountContext="" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409151 4543 reconstruct.go:97] "Volume reconstruction finished" Nov 03 08:39:36.410230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.409159 4543 reconciler.go:29] "Reconciler: start to sync state" Nov 03 08:39:36.411766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.411528 4543 factory.go:153] Registering CRI-O factory Nov 03 08:39:36.411766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.411544 4543 factory.go:223] Registration of the crio container factory successfully Nov 03 08:39:36.411766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.411562 4543 factory.go:103] Registering Raw factory Nov 03 08:39:36.411766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.411576 4543 manager.go:1199] Started watching for new ooms in manager Nov 03 08:39:36.411962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.411940 4543 manager.go:319] Starting recovery of all containers Nov 03 08:39:36.429097 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.429073 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:36.429231 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.429200 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:36.441951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.441913 4543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 08:39:36.443478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.443460 4543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 08:39:36.443478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.443477 4543 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 08:39:36.443633 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.443493 4543 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 08:39:36.443633 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.443526 4543 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 08:39:36.474553 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.474530 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:39:36.474936 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.474901 4543 generic.go:356] "Generic (PLEG): container finished" podID="27e5037f-1bbd-4170-8108-6a06e3f782f2" containerID="c5a38d5da4bd4d26b50666b5c3b0ca99ac2d0bd3e7d55a12eba7b930d5ef17e2" exitCode=1 Nov 03 08:39:36.479913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479887 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="878eeba4410bed3b2b74897062ecf4f58b2fc2b1d4c9371f1ee8471fc088a9d9" exitCode=0 Nov 03 08:39:36.480015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479916 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="44db2e456eaab305387eb9e73c44ccabd839b608371e12436754c704353e74c5" exitCode=0 Nov 03 08:39:36.480015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479930 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="bd4f745079caf45b72868108abfcd29843d49e522034a1aa853f7de4458b2ee0" exitCode=0 Nov 03 08:39:36.480015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479940 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="39e4bc03e41e2280522881311b72eaa073d3aba534f6698f0df45566bf71f59f" exitCode=0 Nov 03 08:39:36.480015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479947 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="723aed93d1e6461be60428cbb04051bf417a8272a581ce6b55cb52bf5598bca1" exitCode=0 Nov 03 08:39:36.480015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.479956 4543 generic.go:356] "Generic (PLEG): container finished" podID="15fbcb5f-5b60-488c-929a-e5661647fb3a" containerID="049c4c36c3c08c753488e6d0448f445e93e226ff2ffb64ddbe80a44d4062d300" exitCode=0 Nov 03 08:39:36.481726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.481700 4543 generic.go:356] "Generic (PLEG): container finished" podID="050f26e6d5e63142521b2810eac72bf7" containerID="2e5874725d3854eea58c260613abe6491202477efb7f5704fd290a3f944591c4" exitCode=0 Nov 03 08:39:36.483088 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.483075 4543 manager.go:324] Recovery completed Nov 03 08:39:36.495998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.495982 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.497070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.497055 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.497158 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.497081 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.497158 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.497092 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.500669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500649 4543 cpu_manager.go:226] "Starting" policy="none" Nov 03 08:39:36.500669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500667 4543 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 08:39:36.500783 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500669 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.500783 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500686 4543 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 08:39:36.500850 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500836 4543 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 08:39:36.500882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500847 4543 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 08:39:36.500882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500864 4543 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 08:39:36.500882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500869 4543 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 08:39:36.500882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500876 4543 policy_none.go:50] "Start" Nov 03 08:39:36.500990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500885 4543 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 08:39:36.500990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.500895 4543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 08:39:36.501069 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501013 4543 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 08:39:36.501069 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501020 4543 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 08:39:36.501069 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501025 4543 policy_none.go:44] "Start" Nov 03 08:39:36.501579 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501564 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.501634 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501596 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.501634 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.501607 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.505374 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.505352 4543 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 08:39:36.506485 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.506471 4543 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:39:36.507975 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.507963 4543 manager.go:341] "Starting Device Plugin manager" Nov 03 08:39:36.508015 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.507986 4543 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 08:39:36.508015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.507995 4543 server.go:87] "Starting device plugin registration server" Nov 03 08:39:36.508310 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508298 4543 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 08:39:36.508341 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508313 4543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 08:39:36.508432 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508411 4543 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 08:39:36.508537 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508506 4543 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 08:39:36.508603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508544 4543 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 08:39:36.508603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508552 4543 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 08:39:36.508707 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.508604 4543 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:39:36.508535105Z","UUID":"85a27f3b-46d8-40f2-a93f-0c1fef522196","Handler":null,"Name":"","Endpoint":""} Nov 03 08:39:36.509750 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.509730 4543 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 08:39:36.509823 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.509781 4543 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" Nov 03 08:39:36.509957 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.509941 4543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 08:39:36.509991 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.509976 4543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 08:39:36.544200 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.544165 4543 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal"] Nov 03 08:39:36.544282 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.544271 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.545226 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.545207 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.545297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.545239 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.545297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.545255 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.545506 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.545496 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.545644 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.545631 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.546403 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546387 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.546403 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546396 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.546491 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546414 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.546491 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546417 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.546491 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546427 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.546491 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546434 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.546605 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.546559 4543 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.546664 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546640 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"20fdfbfab5aa17ccf7324ae585a08734","Type":"ContainerStarted","Data":"1c18fbe5a3943ccfc1839901cc0ed5a866678bb9ba401ad9bdf785ba325232d2"} Nov 03 08:39:36.546702 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546671 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"20fdfbfab5aa17ccf7324ae585a08734","Type":"ContainerStarted","Data":"16baa373627fa1aa0975a496e342ab254a09d02a6ba4ff7d1a8b43b4ee6fc3c4"} Nov 03 08:39:36.546732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546724 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerStarted","Data":"8905cb440dd789eac72ee447160282a4784517333d75ec9c733605dc4cb5ea8c"} Nov 03 08:39:36.546771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546734 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.546771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerDied","Data":"2e5874725d3854eea58c260613abe6491202477efb7f5704fd290a3f944591c4"} Nov 03 08:39:36.546771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.546749 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" event={"ID":"050f26e6d5e63142521b2810eac72bf7","Type":"ContainerStarted","Data":"988c7dc7a0226eb47a11d3f0d2c2becadf184d7e4f7904382a6dfdf5a4f69748"} Nov 03 08:39:36.547498 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.547483 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.547554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.547513 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.547554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.547528 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.547655 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.547644 4543 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-117.us-west-1.compute.internal\" not found" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.596602 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.596578 4543 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 08:39:36.598378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.598364 4543 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:36.598470 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.598455 4543 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:36.606572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.606554 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.608711 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.608696 4543 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 08:39:36.609612 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.609597 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 08:39:36.609681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.609622 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 08:39:36.609681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.609631 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 08:39:36.609681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.609658 4543 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.649188 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.649116 4543 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:36.649188 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.649122 4543 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 08:39:36.689490 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.689471 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:36.689613 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.689598 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.689663 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.689616 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.696840 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.696820 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.696926 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.696845 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.696926 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.696861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.742925 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.742906 4543 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.742983 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.742948 4543 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.743339 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.743327 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeNotReady" Nov 03 08:39:36.743387 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.743352 4543 setters.go:546] "Node became not ready" node="ip-10-0-131-117.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T08:39:36Z","lastTransitionTime":"2026-11-03T08:39:36Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 08:39:36.749406 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.749390 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:36.749456 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:36.749430 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.797977 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.797951 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.798044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.797981 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.798044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.797998 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.798110 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.798058 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.798110 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.798087 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/20fdfbfab5aa17ccf7324ae585a08734-config\") pod \"kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"20fdfbfab5aa17ccf7324ae585a08734\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.798110 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.798058 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/050f26e6d5e63142521b2810eac72bf7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" (UID: \"050f26e6d5e63142521b2810eac72bf7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:36.803789 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:36.803770 4543 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-117.us-west-1.compute.internal" is owned by different node (oldNodeID="8d9faf25-6ee7-4f0b-85cd-129777f7d73a", newNodeID=""), cleaning up... Nov 03 08:39:37.387603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.387577 4543 apiserver.go:52] "Watching apiserver" Nov 03 08:39:37.486718 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.486695 4543 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:37.495023 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.495006 4543 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 08:39:37.495434 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.495417 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j","openshift-ingress-canary/ingress-canary-678m8","openshift-image-registry/node-ca-dfmxv","openshift-network-operator/iptables-alerter-62fp4","openshift-dns/node-resolver-6pr47","openshift-multus/multus-additional-cni-plugins-bj87v","openshift-network-diagnostics/network-check-target-w4gf5","kube-system/global-pull-secret-syncer-qkzjz","openshift-cluster-node-tuning-operator/tuned-f4ldt","kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-d8jcr","openshift-service-ca/service-ca-5fffc97bdf-xsqnh","openshift-insights/insights-runtime-extractor-pd45j","openshift-ovn-kubernetes/ovnkube-node-n26bm","openshift-dns/dns-default-45lmv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal","openshift-multus/multus-4xgnk","kube-system/konnectivity-agent-9rkp2"] Nov 03 08:39:37.495930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.495908 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.500770 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.500734 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:37.506635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.506621 4543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 08:39:37.548198 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.548184 4543 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 08:39:37.548284 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:39:37.548219 4543 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-117.us-west-1.compute.internal" Nov 03 08:39:37.550728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550707 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.550728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550723 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550769 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550784 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550807 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550807 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550837 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.550941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.550912 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\"" Nov 03 08:39:37.551354 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551332 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551614 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551674 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551717 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551736 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551839 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551948 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.551966 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.552133 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.552278 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.552195 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 08:39:37.552744 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.552342 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\"" Nov 03 08:39:37.552744 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.552367 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\"" Nov 03 08:39:37.552744 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.552453 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\"" Nov 03 08:39:37.602949 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.602923 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.603039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.602954 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.603039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.602970 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.603039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.602987 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.603163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603077 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.603163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603096 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.603163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603110 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.603163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603127 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603198 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603227 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b492e70-992c-4975-9153-dbc89048db4b-tmp-dir\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603259 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hl8n\" (UniqueName: \"kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603276 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z86ww\" (UniqueName: \"kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.603342 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603331 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-tmp\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603346 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603371 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603427 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603455 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603495 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603534 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603555 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603570 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.603635 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603618 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603682 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603711 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603738 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603756 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603778 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603792 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603833 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603844 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-cni-binary-copy\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603865 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603914 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.603950 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.604039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604007 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604053 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604085 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvxg\" (UniqueName: \"kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604150 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/530f4286-b8c2-4331-9bcc-372807e290d9-iptables-alerter-script\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604192 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604248 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604255 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npcm\" (UniqueName: \"kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604304 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-data-volume\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c752x\" (UniqueName: \"kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604416 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604471 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604519 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604560 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604604 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604645 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.604678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604715 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604733 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604750 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604780 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604838 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604884 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604903 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604924 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604951 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx4w\" (UniqueName: \"kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604974 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.604979 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b8977ebc-ddf6-41df-9faf-175c92054114-etc-tuned\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605038 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605062 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605099 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605114 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605130 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctsb\" (UniqueName: \"kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605157 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdr9\" (UniqueName: \"kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.605304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605263 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605474 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f00260d-22ef-44e7-802c-d1b9b9585b61-tmp-dir\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605510 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605588 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605610 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605638 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605671 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2f5\" (UniqueName: \"kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605702 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605704 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605735 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncrg\" (UniqueName: \"kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605758 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605781 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnlg\" (UniqueName: \"kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605851 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605877 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605900 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605923 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.605995 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606038 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2qn\" (UniqueName: \"kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:37.606082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606079 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606104 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606159 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606230 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606246 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606316 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606334 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxx6m\" (UniqueName: \"kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.606705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.606661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a57c4f2b-51b2-43d1-9fa4-052405923764-serviceca\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.650871 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.650804 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 08:39:37.650871 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.650815 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.650871 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.650808 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.650871 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.650815 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 08:39:37.651190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651167 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.651190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651173 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 08:39:37.651344 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651207 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 08:39:37.651344 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651135 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.651344 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651328 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\"" Nov 03 08:39:37.651479 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651383 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 08:39:37.651618 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651602 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651622 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651651 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651659 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651690 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651691 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.651719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651716 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651766 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651879 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651881 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651898 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651907 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651913 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651920 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651936 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651964 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 08:39:37.652044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.651967 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652173 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652174 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652190 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652222 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zgq6p\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652222 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652241 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652247 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652256 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652273 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652282 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652277 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 08:39:37.652477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.652319 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 08:39:37.654307 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654290 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovn-node-metrics-cert\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.654382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654365 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-env-overrides\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.654382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654366 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-cert\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:37.654478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654368 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f61e725e-6128-4977-9ac4-ee0df1feb8db-original-pull-secret\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.654478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654399 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b492e70-992c-4975-9153-dbc89048db4b-metrics-tls\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.654478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654435 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e0527392-1028-47e0-a542-80038e79f6da-konnectivity-ca\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:37.654478 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654441 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e0527392-1028-47e0-a542-80038e79f6da-agent-certs\") pod \"konnectivity-agent-9rkp2\" (UID: \"e0527392-1028-47e0-a542-80038e79f6da\") " pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:39:37.654764 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15fbcb5f-5b60-488c-929a-e5661647fb3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.654896 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/42bbce16-077e-4e80-8085-26d639826e05-signing-cabundle\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.654896 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-config\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.654996 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.654971 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b492e70-992c-4975-9153-dbc89048db4b-config-volume\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.655062 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.655047 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-daemon-config\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.655927 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.655911 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/42bbce16-077e-4e80-8085-26d639826e05-signing-key\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.657166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.657126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.657238 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.657223 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27e5037f-1bbd-4170-8108-6a06e3f782f2-ovnkube-script-lib\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.657271 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.657245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.660802 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.660782 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hl8n\" (UniqueName: \"kubernetes.io/projected/530f4286-b8c2-4331-9bcc-372807e290d9-kube-api-access-2hl8n\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.661609 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.661586 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z86ww\" (UniqueName: \"kubernetes.io/projected/15fbcb5f-5b60-488c-929a-e5661647fb3a-kube-api-access-z86ww\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.661702 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.661586 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c752x\" (UniqueName: \"kubernetes.io/projected/2f00260d-22ef-44e7-802c-d1b9b9585b61-kube-api-access-c752x\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.661758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.661704 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npcm\" (UniqueName: \"kubernetes.io/projected/b8977ebc-ddf6-41df-9faf-175c92054114-kube-api-access-5npcm\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.661801 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.661755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvxg\" (UniqueName: \"kubernetes.io/projected/518c5a4f-9809-452d-8a8d-aea90d5961c5-kube-api-access-kjvxg\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.662206 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.662189 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdr9\" (UniqueName: \"kubernetes.io/projected/7b492e70-992c-4975-9153-dbc89048db4b-kube-api-access-bkdr9\") pod \"dns-default-45lmv\" (UID: \"7b492e70-992c-4975-9153-dbc89048db4b\") " pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:37.662479 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.662461 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctsb\" (UniqueName: \"kubernetes.io/projected/a57c4f2b-51b2-43d1-9fa4-052405923764-kube-api-access-dctsb\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.662979 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.662960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx4w\" (UniqueName: \"kubernetes.io/projected/42bbce16-077e-4e80-8085-26d639826e05-kube-api-access-8rx4w\") pod \"service-ca-5fffc97bdf-xsqnh\" (UID: \"42bbce16-077e-4e80-8085-26d639826e05\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xsqnh" Nov 03 08:39:37.663134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.663113 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkn2\" (UniqueName: \"kubernetes.io/projected/7d926adb-98d1-46bb-a1e9-8624624c0f2f-kube-api-access-jkkn2\") pod \"network-check-target-w4gf5\" (UID: \"7d926adb-98d1-46bb-a1e9-8624624c0f2f\") " pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:37.663214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.663200 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnlg\" (UniqueName: \"kubernetes.io/projected/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kube-api-access-rtnlg\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.701454 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.701415 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncrg\" (UniqueName: \"kubernetes.io/projected/bec20a64-e1e9-49fc-bb02-d62c451121e1-kube-api-access-gncrg\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.701543 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.701494 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2f5\" (UniqueName: \"kubernetes.io/projected/27e5037f-1bbd-4170-8108-6a06e3f782f2-kube-api-access-qk2f5\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.701671 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.701654 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxx6m\" (UniqueName: \"kubernetes.io/projected/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-kube-api-access-fxx6m\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.701951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.701901 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2qn\" (UniqueName: \"kubernetes.io/projected/0ae96a70-69ca-4de6-a9f5-8b7202bffba8-kube-api-access-6r2qn\") pod \"ingress-canary-678m8\" (UID: \"0ae96a70-69ca-4de6-a9f5-8b7202bffba8\") " pod="openshift-ingress-canary/ingress-canary-678m8" Nov 03 08:39:37.706495 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706479 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706552 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706503 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706552 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706517 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706552 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706530 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706552 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706546 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706598 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-sys-fs\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706607 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-log-socket\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-socket-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706609 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-registration-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706623 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706646 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706660 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.706681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706677 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706690 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706697 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-system-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706705 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-multus-certs\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706726 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706731 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-etc-selinux\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706726 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-etc-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706763 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706787 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706802 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706807 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-kubelet-config\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-conf-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706871 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/530f4286-b8c2-4331-9bcc-372807e290d9-host-slash\") pod \"iptables-alerter-62fp4\" (UID: \"530f4286-b8c2-4331-9bcc-372807e290d9\") " pod="openshift-network-operator/iptables-alerter-62fp4" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706872 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706885 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706895 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-kubernetes\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706911 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706937 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.706950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706941 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-systemd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706963 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-conf\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706960 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706985 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706982 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-sys\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.706990 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-var-lib-openvswitch\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707006 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707027 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707031 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-socket-dir-parent\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707050 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707074 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707091 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-systemd\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707098 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-netns\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707127 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707134 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-systemd-units\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707187 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-run\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707187 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysctl-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707216 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707239 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-modprobe-d\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707243 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707258 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-var-lib-kubelet\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707281 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707285 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-run-ovn\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707297 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707318 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707341 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707343 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-node-log\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707376 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f00260d-22ef-44e7-802c-d1b9b9585b61-hosts-file\") pod \"node-resolver-6pr47\" (UID: \"2f00260d-22ef-44e7-802c-d1b9b9585b61\") " pod="openshift-dns/node-resolver-6pr47" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707382 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a57c4f2b-51b2-43d1-9fa4-052405923764-host\") pod \"node-ca-dfmxv\" (UID: \"a57c4f2b-51b2-43d1-9fa4-052405923764\") " pod="openshift-image-registry/node-ca-dfmxv" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707387 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707411 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707415 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-etc-kubernetes\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707434 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-multus-cni-dir\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707436 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707466 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-kubelet\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707445 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-system-cni-dir\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707469 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707501 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-bin\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707525 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707552 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707578 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707601 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-bin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707626 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707630 4543 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707638 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f61e725e-6128-4977-9ac4-ee0df1feb8db-dbus\") pod \"global-pull-secret-syncer-qkzjz\" (UID: \"f61e725e-6128-4977-9ac4-ee0df1feb8db\") " pod="kube-system/global-pull-secret-syncer-qkzjz" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707649 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707628 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-hostroot\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c816c18-0b4b-42a4-95cd-dde80e6e75bd-crio-socket\") pod \"insights-runtime-extractor-pd45j\" (UID: \"1c816c18-0b4b-42a4-95cd-dde80e6e75bd\") " pod="openshift-insights/insights-runtime-extractor-pd45j" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-etc-sysconfig\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707681 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707704 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707701 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707729 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707750 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.707732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707770 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707776 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-host\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707775 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-k8s-cni-cncf-io\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707798 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b8977ebc-ddf6-41df-9faf-175c92054114-lib-modules\") pod \"tuned-f4ldt\" (UID: \"b8977ebc-ddf6-41df-9faf-175c92054114\") " pod="openshift-cluster-node-tuning-operator/tuned-f4ldt" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707818 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-cni-multus\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707830 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-slash\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707860 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707832 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27e5037f-1bbd-4170-8108-6a06e3f782f2-host-cni-netd\") pod \"ovnkube-node-n26bm\" (UID: \"27e5037f-1bbd-4170-8108-6a06e3f782f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707885 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707904 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707927 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707934 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8c823c7a-c6fb-4796-9a85-5f23e6c05759-device-dir\") pod \"aws-ebs-csi-driver-node-k285j\" (UID: \"8c823c7a-c6fb-4796-9a85-5f23e6c05759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k285j" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707940 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-run-netns\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707940 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-cnibin\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707977 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707950 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15fbcb5f-5b60-488c-929a-e5661647fb3a-cnibin\") pod \"multus-additional-cni-plugins-bj87v\" (UID: \"15fbcb5f-5b60-488c-929a-e5661647fb3a\") " pod="openshift-multus/multus-additional-cni-plugins-bj87v" Nov 03 08:39:37.709596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.707992 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/518c5a4f-9809-452d-8a8d-aea90d5961c5-host-var-lib-kubelet\") pod \"multus-4xgnk\" (UID: \"518c5a4f-9809-452d-8a8d-aea90d5961c5\") " pod="openshift-multus/multus-4xgnk" Nov 03 08:39:37.710237 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.709893 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bec20a64-e1e9-49fc-bb02-d62c451121e1-metrics-certs\") pod \"network-metrics-daemon-d8jcr\" (UID: \"bec20a64-e1e9-49fc-bb02-d62c451121e1\") " pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.773203 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.773180 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:37.797265 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.797240 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d8jcr" Nov 03 08:39:37.799966 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.799950 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:39:37.900513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:37.900489 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:38.210726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.210699 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d8jcr"] Nov 03 08:39:38.213593 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:38.213568 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec20a64_e1e9_49fc_bb02_d62c451121e1.slice/crio-35527f6070653445e6b8700391f092d7caff935910fb3d5985beabe390d564f0 WatchSource:0}: Error finding container 35527f6070653445e6b8700391f092d7caff935910fb3d5985beabe390d564f0: Status 404 returned error can't find the container with id 35527f6070653445e6b8700391f092d7caff935910fb3d5985beabe390d564f0 Nov 03 08:39:38.362367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.362340 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:38.489913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.489847 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8jcr" event={"ID":"bec20a64-e1e9-49fc-bb02-d62c451121e1","Type":"ContainerStarted","Data":"35527f6070653445e6b8700391f092d7caff935910fb3d5985beabe390d564f0"} Nov 03 08:39:38.572716 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.572692 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:38.908182 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.908132 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:38.972401 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:38.972358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:39.175583 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.175552 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:39.366653 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.366618 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:39.499234 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.497565 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lmv" event={"ID":"7b492e70-992c-4975-9153-dbc89048db4b","Type":"ContainerStarted","Data":"383401d37abcfb37ddd35a42116cb85f1e29f71793bde2e8605c334818fdb1d2"} Nov 03 08:39:39.499234 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.497639 4543 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 08:39:39.499234 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.497658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lmv" event={"ID":"7b492e70-992c-4975-9153-dbc89048db4b","Type":"ContainerStarted","Data":"633722006b2c28da9a7e729abc4314526dfa0a59aea8341b6b3ef9713d6e5e07"} Nov 03 08:39:39.772814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:39.772730 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:40.369858 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.369837 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:40.502404 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.502338 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8jcr" event={"ID":"bec20a64-e1e9-49fc-bb02-d62c451121e1","Type":"ContainerStarted","Data":"d63c422f442a170e560c9d11f7c9adfd86390e9bba3859cfbadf46bda763776e"} Nov 03 08:39:40.502404 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.502394 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d8jcr" event={"ID":"bec20a64-e1e9-49fc-bb02-d62c451121e1","Type":"ContainerStarted","Data":"13ac6578299ea90b482af6969bdb7038c37f7a3a94a4dc7ff21972077a50f226"} Nov 03 08:39:40.504404 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.504381 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pd45j" event={"ID":"1c816c18-0b4b-42a4-95cd-dde80e6e75bd","Type":"ContainerStarted","Data":"55ffc085b923d3843d5b71ff1cd81b39bdc88b5c3f0db7f8a558f6c850847a34"} Nov 03 08:39:40.792056 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.791961 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:40.968248 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:40.968205 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:41.164017 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:41.163976 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:41.570521 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:41.570454 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:41.767036 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:41.766902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:41.966052 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:41.966027 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:44.731898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:44.731859 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:39:45.076982 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:45.076910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:45.859552 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:45.859516 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:46.220914 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:46.220886 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:46.359133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:46.359109 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:46.982190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:46.982165 4543 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-117.us-west-1.compute.internal" event="NodeReady" Nov 03 08:39:47.701565 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.701530 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69656f6bbf-4j7qp"] Nov 03 08:39:47.703306 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.703289 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.769890 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.769861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-ca-trust-extracted\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.769905 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-tls\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.769937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-certificates\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.770001 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-bound-sa-token\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770205 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.770177 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-image-registry-private-configuration\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.770221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh5n\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-kube-api-access-zxh5n\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770315 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.770291 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-trusted-ca\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.770368 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.770334 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-installation-pull-secrets\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.859181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859130 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 08:39:47.859181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859176 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 08:39:47.859409 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859314 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 08:39:47.859409 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859382 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdf8f\"" Nov 03 08:39:47.859484 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859413 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 08:39:47.859972 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.859952 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69656f6bbf-4j7qp"] Nov 03 08:39:47.870661 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-image-registry-private-configuration\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870672 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh5n\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-kube-api-access-zxh5n\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870691 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-trusted-ca\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870708 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-installation-pull-secrets\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870744 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-ca-trust-extracted\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870761 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-tls\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870775 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-certificates\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.870879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.870790 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-bound-sa-token\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.871370 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.871349 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-ca-trust-extracted\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.871614 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.871590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-certificates\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.872005 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.871985 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-trusted-ca\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.873463 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.873444 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-installation-pull-secrets\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.873692 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.873664 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-image-registry-private-configuration\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.873986 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.873701 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-registry-tls\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.888211 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.888191 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:47.931175 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.931129 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-bound-sa-token\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:47.931391 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:47.931373 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh5n\" (UniqueName: \"kubernetes.io/projected/d06ec521-31d5-4f36-ae0e-27efc1c0a1dd-kube-api-access-zxh5n\") pod \"image-registry-69656f6bbf-4j7qp\" (UID: \"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd\") " pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:48.012978 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.012932 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:48.018915 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.018894 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:48.321886 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.321822 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:48.451456 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.451428 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:48.483684 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.483660 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69656f6bbf-4j7qp"] Nov 03 08:39:48.487567 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:48.487545 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06ec521_31d5_4f36_ae0e_27efc1c0a1dd.slice/crio-7739cd43ec7b58e7e19b610c47a33aa7f59043d30accf133bf075276089d141a WatchSource:0}: Error finding container 7739cd43ec7b58e7e19b610c47a33aa7f59043d30accf133bf075276089d141a: Status 404 returned error can't find the container with id 7739cd43ec7b58e7e19b610c47a33aa7f59043d30accf133bf075276089d141a Nov 03 08:39:48.526096 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:48.526075 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" event={"ID":"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd","Type":"ContainerStarted","Data":"7739cd43ec7b58e7e19b610c47a33aa7f59043d30accf133bf075276089d141a"} Nov 03 08:39:49.530207 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:49.530172 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" event={"ID":"d06ec521-31d5-4f36-ae0e-27efc1c0a1dd","Type":"ContainerStarted","Data":"2837ab712b712df2d46e8df53b9644294c0584c8039f50792f42fa9369cb3ac0"} Nov 03 08:39:49.530625 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:49.530309 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:39:49.700935 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:49.700867 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" podStartSLOduration=2.700836026 podStartE2EDuration="2.700836026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:49.700775089 +0000 UTC m=+13.368815829" watchObservedRunningTime="2026-11-03 08:39:49.700836026 +0000 UTC m=+13.368876767" Nov 03 08:39:49.794510 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:49.794454 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45lmv" Nov 03 08:39:50.408033 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:50.407998 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:50.538043 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:50.538014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:50.665767 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:50.665702 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:50.849435 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:50.849407 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:55.455428 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:55.455402 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n26bm" Nov 03 08:39:55.708570 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:55.708496 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:56.715189 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:56.715156 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:56.943811 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:56.943777 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.194747 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.194714 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.326125 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.326095 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.407978 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.407947 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt"] Nov 03 08:39:57.410945 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.410927 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.427606 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.427586 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.427722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.427631 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.427722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.427651 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhq9\" (UniqueName: \"kubernetes.io/projected/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-kube-api-access-8qhq9\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.427841 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.427783 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.456354 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.456302 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.516202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.516176 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 08:39:57.516330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.516207 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6ph5f\"" Nov 03 08:39:57.516330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.516214 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 08:39:57.516921 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.516903 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt"] Nov 03 08:39:57.518907 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.518873 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 08:39:57.519009 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.518954 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 08:39:57.519009 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.518984 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 08:39:57.519825 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.519807 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9"] Nov 03 08:39:57.521705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.521689 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.528256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.528238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.528336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.528265 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhq9\" (UniqueName: \"kubernetes.io/projected/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-kube-api-access-8qhq9\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.528336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.528284 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.528336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.528312 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.528926 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.528906 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.530883 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.530861 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.530970 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.530913 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.571267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.571243 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ttb6z"] Nov 03 08:39:57.573534 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.573519 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.576322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.576305 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 08:39:57.576405 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.576387 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pvjdx\"" Nov 03 08:39:57.576469 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.576440 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 08:39:57.576555 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.576543 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 08:39:57.581790 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.581771 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9"] Nov 03 08:39:57.587596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.587575 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhq9\" (UniqueName: \"kubernetes.io/projected/6eff5b5c-c34c-422b-9a8e-ce973b09c18c-kube-api-access-8qhq9\") pod \"openshift-state-metrics-7bd7489957-f6spt\" (UID: \"6eff5b5c-c34c-422b-9a8e-ce973b09c18c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.602180 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.602158 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.628772 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.628752 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2tx\" (UniqueName: \"kubernetes.io/projected/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-api-access-hp2tx\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.628867 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.628778 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.628867 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.628797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.628867 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.628815 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-tls\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.628965 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.628896 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.629045 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629030 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-kube-api-access-x8z4w\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629066 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-metrics-client-ca\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629114 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629084 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.629114 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629105 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-root\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629228 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629155 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629263 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629248 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-textfile\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.629297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629292 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.629380 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629315 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-wtmp\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.629380 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.629369 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-sys\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.677125 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.677101 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h97f4\"" Nov 03 08:39:57.677262 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.677248 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 08:39:57.677312 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.677264 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 08:39:57.677312 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.677286 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 08:39:57.722591 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.722540 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" Nov 03 08:39:57.730418 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730398 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.730492 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730435 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.730492 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730478 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-wtmp\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.730586 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730507 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-sys\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.730586 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730535 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2tx\" (UniqueName: \"kubernetes.io/projected/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-api-access-hp2tx\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.730677 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730622 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-sys\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.730677 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-wtmp\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.730758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730699 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.730816 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730737 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.730816 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730795 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-tls\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730854 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.731037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730896 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-kube-api-access-x8z4w\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.730959 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-metrics-client-ca\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731001 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.731037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731027 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-root\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731057 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731109 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-textfile\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731244 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.731796 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731548 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-root\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.731796 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731602 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.731999 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.731886 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.732115 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.732094 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-textfile\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.732807 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.732778 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-metrics-client-ca\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.733571 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.733164 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.733817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.733793 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.734399 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.734374 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.734549 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.734532 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.735247 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.735209 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-node-exporter-tls\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.790880 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.790850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2tx\" (UniqueName: \"kubernetes.io/projected/9edb5966-dc9a-4536-bfa9-bcfe3df2baf1-kube-api-access-hp2tx\") pod \"kube-state-metrics-5dc8d7d89b-wqht9\" (UID: \"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.791235 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.791218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z4w\" (UniqueName: \"kubernetes.io/projected/13dadf9a-ed46-4f0e-8bbc-5925697bb5f8-kube-api-access-x8z4w\") pod \"node-exporter-ttb6z\" (UID: \"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8\") " pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.834374 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.834349 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" Nov 03 08:39:57.886977 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.886947 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ttb6z" Nov 03 08:39:57.897517 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:57.897435 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:57.904163 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:57.903681 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dadf9a_ed46_4f0e_8bbc_5925697bb5f8.slice/crio-e90200ef2e53196d35f9cb24d1382afe5dddbccbd86d6a835a539bc78b110732 WatchSource:0}: Error finding container e90200ef2e53196d35f9cb24d1382afe5dddbccbd86d6a835a539bc78b110732: Status 404 returned error can't find the container with id e90200ef2e53196d35f9cb24d1382afe5dddbccbd86d6a835a539bc78b110732 Nov 03 08:39:58.092346 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.092279 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt"] Nov 03 08:39:58.095098 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:58.095073 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eff5b5c_c34c_422b_9a8e_ce973b09c18c.slice/crio-bdd6da02a20c97528947bf1d51f653831e9b1f9b9b9b8d934ccc0c4d0f8634e9 WatchSource:0}: Error finding container bdd6da02a20c97528947bf1d51f653831e9b1f9b9b9b8d934ccc0c4d0f8634e9: Status 404 returned error can't find the container with id bdd6da02a20c97528947bf1d51f653831e9b1f9b9b9b8d934ccc0c4d0f8634e9 Nov 03 08:39:58.115936 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.115911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:58.204562 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.204540 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9"] Nov 03 08:39:58.207579 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:39:58.207553 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edb5966_dc9a_4536_bfa9_bcfe3df2baf1.slice/crio-e8835c42bee93021daad26f034039b57a7ed18cff77758c70f39ae6346a5d480 WatchSource:0}: Error finding container e8835c42bee93021daad26f034039b57a7ed18cff77758c70f39ae6346a5d480: Status 404 returned error can't find the container with id e8835c42bee93021daad26f034039b57a7ed18cff77758c70f39ae6346a5d480 Nov 03 08:39:58.563900 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.563865 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttb6z" event={"ID":"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8","Type":"ContainerStarted","Data":"e90200ef2e53196d35f9cb24d1382afe5dddbccbd86d6a835a539bc78b110732"} Nov 03 08:39:58.565040 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.565015 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" event={"ID":"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1","Type":"ContainerStarted","Data":"e8835c42bee93021daad26f034039b57a7ed18cff77758c70f39ae6346a5d480"} Nov 03 08:39:58.566810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.566790 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" event={"ID":"6eff5b5c-c34c-422b-9a8e-ce973b09c18c","Type":"ContainerStarted","Data":"8ebcb99ea3f0adef7b480b63f7610f465c376f72fe7ca6c4182969ec6f8b80ee"} Nov 03 08:39:58.566889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.566837 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" event={"ID":"6eff5b5c-c34c-422b-9a8e-ce973b09c18c","Type":"ContainerStarted","Data":"e29174ca76dc86dc0381ab7cc5b210a89a9fdb57892d904952745778023e4734"} Nov 03 08:39:58.566889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.566877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" event={"ID":"6eff5b5c-c34c-422b-9a8e-ce973b09c18c","Type":"ContainerStarted","Data":"bdd6da02a20c97528947bf1d51f653831e9b1f9b9b9b8d934ccc0c4d0f8634e9"} Nov 03 08:39:58.836335 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:58.836220 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:59.113611 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:59.113552 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:59.253029 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:59.253001 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:59.734013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:59.733971 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:39:59.867917 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:39:59.867888 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:00.579710 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:00.579675 4543 generic.go:356] "Generic (PLEG): container finished" podID="13dadf9a-ed46-4f0e-8bbc-5925697bb5f8" containerID="34b3526dffc9ecf514e03f763de1a721a38520ad430d153502ce05cb0222f2c8" exitCode=0 Nov 03 08:40:00.579815 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:00.579774 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttb6z" event={"ID":"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8","Type":"ContainerDied","Data":"34b3526dffc9ecf514e03f763de1a721a38520ad430d153502ce05cb0222f2c8"} Nov 03 08:40:00.581372 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:00.581342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" event={"ID":"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1","Type":"ContainerStarted","Data":"40196227f66bae64df406f5bc3a0414cefd48155efcbf994ef8641aab265843a"} Nov 03 08:40:00.586681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:00.586656 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" event={"ID":"6eff5b5c-c34c-422b-9a8e-ce973b09c18c","Type":"ContainerStarted","Data":"759dc5f20ff46fd28b9f64364d9f15b1b42b289ca779b17f7f270fc42c6c16ab"} Nov 03 08:40:00.867470 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:00.867422 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-f6spt" podStartSLOduration=1.9636343809999999 podStartE2EDuration="3.867407421s" totalImagesPullingTime="1.90377304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:00.866594558 +0000 UTC m=+24.534635296" watchObservedRunningTime="2026-11-03 08:40:00.867407421 +0000 UTC m=+24.535448159" Nov 03 08:40:01.591936 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.591901 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttb6z" event={"ID":"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8","Type":"ContainerStarted","Data":"f50e6cc7247b6397ea80439ffa1b0b1fc11c264a4b71ac90c088ee67ea1ecc35"} Nov 03 08:40:01.592378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.591949 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ttb6z" event={"ID":"13dadf9a-ed46-4f0e-8bbc-5925697bb5f8","Type":"ContainerStarted","Data":"6c5b7304481c556c476318ed2b1f0573156d47c8bb879f6b5acea1c70b9766f8"} Nov 03 08:40:01.594132 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.594107 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" event={"ID":"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1","Type":"ContainerStarted","Data":"2f1f0db0cdb79f7f3d51d9e4177945ecbdff4bf891aefd9c7804d53f363b9041"} Nov 03 08:40:01.594243 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.594169 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" event={"ID":"9edb5966-dc9a-4536-bfa9-bcfe3df2baf1","Type":"ContainerStarted","Data":"7880c2aa9c59477b83d7d585715d1f19b41168db30475900aad6868f7f17892b"} Nov 03 08:40:01.711010 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.710966 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ttb6z" podStartSLOduration=2.730638543 podStartE2EDuration="4.710953603s" totalImagesPullingTime="1.887180606s" totalInitContainerRuntime="93.134454ms" isStatefulPod=true podCreationTimestamp="2026-11-03 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:01.710183427 +0000 UTC m=+25.378224165" watchObservedRunningTime="2026-11-03 08:40:01.710953603 +0000 UTC m=+25.378994340" Nov 03 08:40:01.828897 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.828281 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-wqht9" podStartSLOduration=2.590565591 podStartE2EDuration="4.828262454s" totalImagesPullingTime="2.237696863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:01.827596289 +0000 UTC m=+25.495637027" watchObservedRunningTime="2026-11-03 08:40:01.828262454 +0000 UTC m=+25.496303194" Nov 03 08:40:01.973277 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:01.973243 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:02.103449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:02.103421 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:02.243301 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:02.243206 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:02.390011 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:02.389954 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:03.612249 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:03.612212 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:03.768469 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:03.768437 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:03.953902 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:03.953829 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:04.076098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:04.076072 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rkp2" Nov 03 08:40:04.735023 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:04.734988 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w4gf5" Nov 03 08:40:05.558812 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:05.558776 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:40:05.888088 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:05.888049 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.087632 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.087608 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.229990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.229930 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.366525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.366503 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.547314 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.547249 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-7gbhp"] Nov 03 08:40:06.582407 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.582385 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.608367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.608342 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7gbhp"] Nov 03 08:40:06.639274 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.639255 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-wks45\"" Nov 03 08:40:06.657807 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.657785 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 08:40:06.657891 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.657806 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 08:40:06.701195 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.701172 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-tmp\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.701275 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.701210 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klgk9\" (UniqueName: \"kubernetes.io/projected/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-kube-api-access-klgk9\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.751917 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.751894 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.801888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.801822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-tmp\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.801888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.801872 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klgk9\" (UniqueName: \"kubernetes.io/projected/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-kube-api-access-klgk9\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.802266 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.802246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-tmp\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.868803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.868773 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klgk9\" (UniqueName: \"kubernetes.io/projected/e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8-kube-api-access-klgk9\") pod \"downloads-6dd789c8b7-7gbhp\" (UID: \"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8\") " pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.906915 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.906894 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:06.945849 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.945826 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:06.950295 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.950275 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:06.971414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:06.971387 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.003958 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.003935 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.003963 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.003984 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004001 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004034 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004049 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004068 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqj8r\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004083 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004097 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004112 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004132 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004174 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004189 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004202 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004223 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004238 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004262 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004277 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.004340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.004292 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.020053 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.020029 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:07.026885 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.026865 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 08:40:07.026985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.026907 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 08:40:07.026985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.026908 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 08:40:07.026985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.026928 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 08:40:07.068842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068789 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 08:40:07.068946 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068846 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 08:40:07.068946 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068848 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dwnnn\"" Nov 03 08:40:07.068946 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068890 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 08:40:07.068946 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068930 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3lbj6v94lpqq8\"" Nov 03 08:40:07.069105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068973 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 08:40:07.069105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.068988 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 08:40:07.069105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.069003 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 08:40:07.104754 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104730 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104757 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104774 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104792 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104812 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104828 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.104854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104852 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqj8r\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104906 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104951 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.104976 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105012 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105035 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105056 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105088 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105179 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105205 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105387 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.105634 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.107443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.106783 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.107443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.107393 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108563 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108733 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108709 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108808 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108749 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108866 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108815 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.108951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.108929 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.109159 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.109117 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.109812 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.109794 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.110330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.110310 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.110330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.110317 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.127003 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.126986 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 08:40:07.128202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.128180 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 08:40:07.128305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.128216 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 08:40:07.128305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.128243 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 08:40:07.128305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.128268 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 08:40:07.136052 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.136036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.136155 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.136085 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.137205 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.137190 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.138313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.138295 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.140100 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.140077 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.172234 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.172214 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqj8r\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\") pod \"prometheus-k8s-0\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.343313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.343257 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-7gbhp"] Nov 03 08:40:07.345904 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:40:07.345873 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6de53d9_dec4_4b9a_a826_0d3a3e8d37f8.slice/crio-a5782c7746380829872dcf5622088e57bf3d448b0f1106887b532371eca91774 WatchSource:0}: Error finding container a5782c7746380829872dcf5622088e57bf3d448b0f1106887b532371eca91774: Status 404 returned error can't find the container with id a5782c7746380829872dcf5622088e57bf3d448b0f1106887b532371eca91774 Nov 03 08:40:07.360087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.360066 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:07.382307 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.382291 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:07.534623 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.534599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:07.614623 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.614599 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7gbhp" event={"ID":"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8","Type":"ContainerStarted","Data":"a5782c7746380829872dcf5622088e57bf3d448b0f1106887b532371eca91774"} Nov 03 08:40:07.840265 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:07.840193 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:07.842629 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:40:07.842610 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c000448_3f04_4c75_a991_5cc9fc3dd19a.slice/crio-bdc03263b427fcc77ac4dcd5fdf4ef63029da9ed3b518b8df90fb1fec0e81b45 WatchSource:0}: Error finding container bdc03263b427fcc77ac4dcd5fdf4ef63029da9ed3b518b8df90fb1fec0e81b45: Status 404 returned error can't find the container with id bdc03263b427fcc77ac4dcd5fdf4ef63029da9ed3b518b8df90fb1fec0e81b45 Nov 03 08:40:08.215861 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:08.215827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:08.377113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:08.377078 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:08.619577 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:08.619477 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"bdc03263b427fcc77ac4dcd5fdf4ef63029da9ed3b518b8df90fb1fec0e81b45"} Nov 03 08:40:10.540887 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:10.540858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69656f6bbf-4j7qp" Nov 03 08:40:10.628490 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:10.628457 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" exitCode=0 Nov 03 08:40:10.628662 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:10.628500 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} Nov 03 08:40:10.838181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:10.838089 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:10.976486 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:10.976453 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:11.157951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:11.157915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:11.355529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:11.355492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:13.948595 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:13.948550 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:14.083306 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:14.083266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:14.657157 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:14.657103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} Nov 03 08:40:14.657315 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:14.657210 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} Nov 03 08:40:17.088897 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.088865 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:17.263665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.263583 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:17.673533 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.673498 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} Nov 03 08:40:17.673699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.673605 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} Nov 03 08:40:17.673699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.673635 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} Nov 03 08:40:17.673699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.673659 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerStarted","Data":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} Nov 03 08:40:17.899068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.899011 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.849060914 podStartE2EDuration="11.898995517s" totalImagesPullingTime="8.027255226s" totalInitContainerRuntime="22.679377ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:17.898716946 +0000 UTC m=+41.566757685" watchObservedRunningTime="2026-11-03 08:40:17.898995517 +0000 UTC m=+41.567036246" Nov 03 08:40:17.942055 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:17.941976 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:18.080369 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:18.080335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:21.993355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:21.993296 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:22.126651 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:22.126616 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:22.382913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:22.382886 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:23.162622 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.162585 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:23.290873 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.290813 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:23.697219 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.697181 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-7gbhp" event={"ID":"e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8","Type":"ContainerStarted","Data":"bac856c25a4cdf8cbafd81f823847bc3fa9b8c9701de67aeb41d7d8b62730f30"} Nov 03 08:40:23.697382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.697366 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:23.698519 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.698497 4543 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-7gbhp container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" start-of-body= Nov 03 08:40:23.698606 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.698537 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-7gbhp" podUID="e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.13:8080/\": dial tcp 10.134.0.13:8080: connect: connection refused" Nov 03 08:40:23.852170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:23.852053 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-7gbhp" podStartSLOduration=1.605508125 podStartE2EDuration="17.852034671s" totalImagesPullingTime="16.246526546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:23.850961211 +0000 UTC m=+47.519001949" watchObservedRunningTime="2026-11-03 08:40:23.852034671 +0000 UTC m=+47.520075408" Nov 03 08:40:24.717866 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:24.717828 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-7gbhp" Nov 03 08:40:27.994847 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:27.994799 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:28.128008 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:28.127975 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:30.413490 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:30.413458 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:30.461183 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:30.461155 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:30.747661 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:30.747581 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:31.218419 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:31.218373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:31.349682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:31.349648 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:32.960426 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:32.960397 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:34.968461 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:34.968423 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:35.100746 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:35.100723 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:35.560348 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:35.560319 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:40:42.570096 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:42.570065 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:42.700589 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:42.700565 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:46.219821 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:46.219783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:46.349611 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:46.349583 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:49.013879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:49.013846 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:49.154801 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:49.154759 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:49.289655 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:49.289585 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:49.482266 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:49.482238 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:57.410256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410222 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:57.410759 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410737 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="prometheus" containerID="cri-o://94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" gracePeriod=600 Nov 03 08:40:57.410831 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410798 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="thanos-sidecar" containerID="cri-o://dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" gracePeriod=600 Nov 03 08:40:57.410890 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410833 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" gracePeriod=600 Nov 03 08:40:57.410890 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410829 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-web" containerID="cri-o://71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" gracePeriod=600 Nov 03 08:40:57.410890 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410847 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="config-reloader" containerID="cri-o://292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" gracePeriod=600 Nov 03 08:40:57.411017 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.410832 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy" containerID="cri-o://92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" gracePeriod=600 Nov 03 08:40:57.698795 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.698771 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:57.798017 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.797991 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798169 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798026 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798169 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798047 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798169 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798066 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798238 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798328 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798295 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798375 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798329 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798375 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798361 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798482 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798390 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798482 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798426 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798482 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798439 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.798639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798482 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798521 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798551 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798580 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798610 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798893 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798644 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798893 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798674 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.798893 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798706 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.799042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798932 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.799042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798967 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\") pod \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\" (UID: \"7c000448-3f04-4c75-a991-5cc9fc3dd19a\") " Nov 03 08:40:57.799042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.798962 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.799233 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.799216 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.799305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.799237 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.799869 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.799585 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.799990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.799931 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.800412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.799490 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.801365 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801117 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.801521 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801496 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:40:57.801729 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801711 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.801842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801736 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "kube-api-access-wqj8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:57.802078 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:40:57.802192 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801760 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.802192 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.801832 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.802358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.802215 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:40:57.802889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.802856 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.802998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.802882 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.803726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.803700 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.803942 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.803925 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.804370 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.804348 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.804457 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.804378 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:40:57.813052 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.813026 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config" pod "7c000448-3f04-4c75-a991-5cc9fc3dd19a" (UID: "7c000448-3f04-4c75-a991-5cc9fc3dd19a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:40:57.818547 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818526 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" exitCode=0 Nov 03 08:40:57.818547 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818544 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" exitCode=0 Nov 03 08:40:57.818665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818552 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" exitCode=0 Nov 03 08:40:57.818665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818558 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" exitCode=0 Nov 03 08:40:57.818665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818566 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" exitCode=0 Nov 03 08:40:57.818665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818572 4543 generic.go:356] "Generic (PLEG): container finished" podID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" exitCode=0 Nov 03 08:40:57.818665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818593 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818664 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818670 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818697 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818721 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818744 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818750 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818768 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} Nov 03 08:40:57.818806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.818793 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7c000448-3f04-4c75-a991-5cc9fc3dd19a","Type":"ContainerDied","Data":"bdc03263b427fcc77ac4dcd5fdf4ef63029da9ed3b518b8df90fb1fec0e81b45"} Nov 03 08:40:57.828094 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.827252 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.834212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.834194 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.841616 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.841597 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.848308 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.848291 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.856103 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.856084 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.862960 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.862945 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.869262 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.869244 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.869539 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.869519 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.869590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.869547 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.869590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.869583 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.869806 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.869792 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.869859 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.869809 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.869859 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.869821 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.870027 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.870011 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.870090 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870032 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.870090 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870046 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.870298 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.870283 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.870339 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870305 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.870339 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870324 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.870514 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.870501 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.870556 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870518 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.870556 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870532 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.870715 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.870702 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.870757 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870718 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.870757 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870730 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.870982 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:40:57.870965 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.871026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.870989 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.871026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871004 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.871229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871212 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.871270 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871229 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.871439 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871424 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.871483 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871438 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.871640 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871625 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.871678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871642 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.871830 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871810 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.871875 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871830 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.872001 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.871980 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.872064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872002 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.872247 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872229 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.872286 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872247 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.872442 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872427 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.872489 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872450 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.872682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872666 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.872733 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872682 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.872888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872873 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.872888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.872888 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.873124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873105 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.873215 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873125 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.873394 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873376 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.873451 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873395 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.873640 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873619 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.873719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873641 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.873886 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873869 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.873989 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.873886 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.874096 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874081 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.874179 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874098 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.874312 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874296 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.874354 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874312 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.874510 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874494 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.874551 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874510 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.874737 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874718 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.874737 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874737 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.874978 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874951 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.874978 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.874970 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.875196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875180 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.875330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875195 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.875418 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875400 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.875462 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875419 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.875638 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875617 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.875689 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875638 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.875853 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875837 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.875898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.875854 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.876072 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876055 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.876118 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876073 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.876323 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876306 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.876374 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876325 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.876568 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876550 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.876615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876568 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.876782 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876767 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.876822 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876782 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.876988 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876973 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.877041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.876988 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.877232 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877216 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.877232 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877232 4543 scope.go:122] "RemoveContainer" containerID="fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855" Nov 03 08:40:57.877477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877460 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855"} err="failed to get container status \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": rpc error: code = NotFound desc = could not find container \"fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855\": container with ID starting with fa8117c110b99f0eb8bfb1e3d291de3fbbae77173cf4aaba72eaaebdc485a855 not found: ID does not exist" Nov 03 08:40:57.877525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877478 4543 scope.go:122] "RemoveContainer" containerID="92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f" Nov 03 08:40:57.877692 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877675 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f"} err="failed to get container status \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": rpc error: code = NotFound desc = could not find container \"92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f\": container with ID starting with 92f1f7b858930902caed97fad1f28982ba87f97e823b0a7fc2cc715b30dba06f not found: ID does not exist" Nov 03 08:40:57.877735 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877692 4543 scope.go:122] "RemoveContainer" containerID="71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44" Nov 03 08:40:57.877898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877880 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44"} err="failed to get container status \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": rpc error: code = NotFound desc = could not find container \"71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44\": container with ID starting with 71cec71ba96d22b1628dee5c06632436bed5ec62103a0882fd91facddd103c44 not found: ID does not exist" Nov 03 08:40:57.877961 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.877899 4543 scope.go:122] "RemoveContainer" containerID="dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de" Nov 03 08:40:57.878112 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878096 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de"} err="failed to get container status \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": rpc error: code = NotFound desc = could not find container \"dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de\": container with ID starting with dd5451d8080f4a99f10ccf9e40febe485ce15f09bef19a0e0fe3245efedd51de not found: ID does not exist" Nov 03 08:40:57.878112 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878111 4543 scope.go:122] "RemoveContainer" containerID="292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc" Nov 03 08:40:57.878334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878317 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc"} err="failed to get container status \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": rpc error: code = NotFound desc = could not find container \"292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc\": container with ID starting with 292a641246475e7658d25a566b1432e600116cf04a3a2e1339bdfc437be607dc not found: ID does not exist" Nov 03 08:40:57.878382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878334 4543 scope.go:122] "RemoveContainer" containerID="94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b" Nov 03 08:40:57.878534 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878513 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b"} err="failed to get container status \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": rpc error: code = NotFound desc = could not find container \"94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b\": container with ID starting with 94a254da54d83a432cab8ffa025eb1ced48eaa1cb0080b0b319ae36ec0e10b7b not found: ID does not exist" Nov 03 08:40:57.878602 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878535 4543 scope.go:122] "RemoveContainer" containerID="a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e" Nov 03 08:40:57.878765 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.878751 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e"} err="failed to get container status \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": rpc error: code = NotFound desc = could not find container \"a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e\": container with ID starting with a3bd18d37974a39cb496c21908e7cef296f941c655a9006fcc1e4371f2297e0e not found: ID does not exist" Nov 03 08:40:57.899959 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.899939 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.899964 4543 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.899979 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.899989 4543 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-grpc-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.899998 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900007 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900015 4543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-config-out\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900023 4543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900032 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqj8r\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-kube-api-access-wqj8r\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900040 4543 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-kube-rbac-proxy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900047 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-db\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900056 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900064 4543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c000448-3f04-4c75-a991-5cc9fc3dd19a-tls-assets\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900071 4543 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900079 4543 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-configmap-metrics-client-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900087 4543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c000448-3f04-4c75-a991-5cc9fc3dd19a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900095 4543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-web-config\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.900388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.900103 4543 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7c000448-3f04-4c75-a991-5cc9fc3dd19a-secret-metrics-client-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:40:57.955927 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.955860 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:57.962305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.962281 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:57.994459 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994432 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:57.994673 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994662 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-thanos" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994675 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-thanos" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994692 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="prometheus" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994697 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="prometheus" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994706 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="init-config-reloader" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994712 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="init-config-reloader" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994718 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="config-reloader" Nov 03 08:40:57.994720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994723 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="config-reloader" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994729 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994734 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994742 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-web" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994747 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-web" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994754 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="thanos-sidecar" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994758 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="thanos-sidecar" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994799 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="config-reloader" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994806 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994812 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-web" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994817 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="kube-rbac-proxy-thanos" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994822 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="thanos-sidecar" Nov 03 08:40:57.994911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:57.994828 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" containerName="prometheus" Nov 03 08:40:58.032880 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.032851 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.079931 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.075324 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:58.088415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.088384 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 08:40:58.088415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.088403 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 08:40:58.088590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.088565 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 08:40:58.088590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.088565 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 08:40:58.101617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101588 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101723 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101623 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101723 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101650 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101811 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101787 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101846 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101825 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.101889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101869 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102001 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.101986 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102049 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102017 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102049 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102174 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102084 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102174 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102117 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102181 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102205 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nmb5\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-kube-api-access-6nmb5\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102259 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102399 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102293 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102454 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102395 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102454 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102423 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102457 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.102545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.102492 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.139264 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139233 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 08:40:58.139264 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139261 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139264 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139317 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139328 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139321 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139320 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 08:40:58.139452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139366 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 08:40:58.139620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139456 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3lbj6v94lpqq8\"" Nov 03 08:40:58.139620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139465 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 08:40:58.139620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139511 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 08:40:58.139620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139590 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dwnnn\"" Nov 03 08:40:58.139727 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.139671 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 08:40:58.203118 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203093 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203121 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203161 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nmb5\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-kube-api-access-6nmb5\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203207 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203246 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203270 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203337 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203363 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203421 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203455 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203480 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.203582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203552 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203615 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203645 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.203959 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204389 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.204365 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.204902 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.204874 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.205829 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.205242 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.205829 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.205535 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.206537 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.206466 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config-out\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.206817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.206788 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.207049 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.207000 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.207331 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.207300 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.207532 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.207509 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.207695 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.207667 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f8043a05-3979-4dd3-9a36-5bf3d14ed924-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.208256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.208230 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.208894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.208875 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.209197 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.209175 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.209293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.209280 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.209442 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.209422 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.209539 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.209521 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.209664 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.209649 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f8043a05-3979-4dd3-9a36-5bf3d14ed924-web-config\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.262621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.262589 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nmb5\" (UniqueName: \"kubernetes.io/projected/f8043a05-3979-4dd3-9a36-5bf3d14ed924-kube-api-access-6nmb5\") pod \"prometheus-k8s-0\" (UID: \"f8043a05-3979-4dd3-9a36-5bf3d14ed924\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.442582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.442546 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:40:58.448800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.448771 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c000448-3f04-4c75-a991-5cc9fc3dd19a" path="/var/lib/kubelet/pods/7c000448-3f04-4c75-a991-5cc9fc3dd19a/volumes" Nov 03 08:40:58.907291 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:58.907258 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 08:40:58.910118 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:40:58.910090 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8043a05_3979_4dd3_9a36_5bf3d14ed924.slice/crio-ddb094aa74db7be141c3be0ded318953cdb9f6513454b7331d70cd1431b6ffe4 WatchSource:0}: Error finding container ddb094aa74db7be141c3be0ded318953cdb9f6513454b7331d70cd1431b6ffe4: Status 404 returned error can't find the container with id ddb094aa74db7be141c3be0ded318953cdb9f6513454b7331d70cd1431b6ffe4 Nov 03 08:40:59.164392 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.164277 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:59.364402 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.364360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:59.597559 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.597462 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:59.816032 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.815950 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:40:59.828280 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.828253 4543 generic.go:356] "Generic (PLEG): container finished" podID="f8043a05-3979-4dd3-9a36-5bf3d14ed924" containerID="2f1286bcea4bc49d1b4b0df2867680a3f14719fb7f1ed4c8a3ed13feb7812c80" exitCode=0 Nov 03 08:40:59.828416 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.828290 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerDied","Data":"2f1286bcea4bc49d1b4b0df2867680a3f14719fb7f1ed4c8a3ed13feb7812c80"} Nov 03 08:40:59.828416 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:40:59.828356 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"ddb094aa74db7be141c3be0ded318953cdb9f6513454b7331d70cd1431b6ffe4"} Nov 03 08:41:00.836322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836239 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"c0f9eca45108681687c2b98a882ad33123c386bb0e5298e6e1b29784ba6de541"} Nov 03 08:41:00.836322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836301 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"d66e35184308f9ea1a222782a59ba1eb27111690225f2da2e511ff7b5ea8388c"} Nov 03 08:41:00.836322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836326 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"c761fed40fe7e7596055914eeda26e46a8b5a09f37f5dd44473afa741306c5b0"} Nov 03 08:41:00.836810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836350 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"c422e80e2b5095a590eb1683d026c9aa84f3757f48a5f6c76d0d54e50c1d0dc6"} Nov 03 08:41:00.836810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836379 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"d1629f7c384fe3fb5dc5665758b976f33675182a31d6bd050e545e5439ccb34c"} Nov 03 08:41:00.836810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.836401 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f8043a05-3979-4dd3-9a36-5bf3d14ed924","Type":"ContainerStarted","Data":"9af761f3d4022cd4d471941f5e9403839314eeebdb825e27a867a74a010de2b5"} Nov 03 08:41:00.963217 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:00.963158 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.948133657 podStartE2EDuration="3.963122725s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.989068ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:00.962123792 +0000 UTC m=+84.630164534" watchObservedRunningTime="2026-11-03 08:41:00.963122725 +0000 UTC m=+84.631163465" Nov 03 08:41:01.217357 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:01.217321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:41:01.351367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:01.351338 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:41:03.443603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:03.443567 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:41:05.560656 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:41:05.560623 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:41:17.489786 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:17.489750 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:41:35.560361 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:41:35.560325 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:41:58.443452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:58.443359 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:41:58.459627 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:58.459600 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:41:59.052286 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:41:59.052262 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 08:42:05.559772 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:42:05.559735 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:42:07.322085 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:07.322047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:07.450943 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:07.450911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:10.616387 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:10.616347 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:11.019859 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:11.019759 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:17.757723 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:17.757687 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:20.058336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:20.058303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:20.186086 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:20.186048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:24.099757 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:24.099720 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:24.225257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:24.225222 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:31.223103 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:31.223065 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:31.351275 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:31.351241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:35.558612 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:42:35.558577 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:42:55.097199 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:55.097169 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:42:55.227017 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:42:55.226987 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:43:05.559010 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:43:05.558974 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:43:18.160187 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:43:18.160115 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:43:35.559869 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:43:35.559830 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:43:36.103288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:43:36.103251 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:05.117481 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:05.117447 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:05.560364 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:44:05.560283 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:44:14.417647 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:14.417614 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:14.550687 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:14.550658 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:18.449476 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:18.449445 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:21.917817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:21.917779 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:22.119748 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:22.119716 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:22.289830 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:22.289731 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:23.467396 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:23.467366 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:23.606256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:23.606223 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:24.748555 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:24.748512 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:29.904125 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:29.904085 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:30.109150 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:30.109107 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:31.013865 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.013831 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:31.328930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.328897 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:31.383324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.383296 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8"] Nov 03 08:44:31.385382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.385365 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.463050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.463011 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:31.470131 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.470108 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.470262 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.470236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcsq\" (UniqueName: \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.470325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.470296 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.489655 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.489627 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 08:44:31.489758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.489680 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 08:44:31.489758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.489681 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7cvpl\"" Nov 03 08:44:31.491004 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.490987 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8"] Nov 03 08:44:31.571322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.571266 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcsq\" (UniqueName: \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.571322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.571303 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.571452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.571432 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.571666 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.571647 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.571729 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.571712 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.631347 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.631321 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcsq\" (UniqueName: \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:31.697001 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:31.696977 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:32.065615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:32.065582 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8"] Nov 03 08:44:32.068841 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:44:32.068814 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7b93cd_3326_48f2_8570_a467476bc6d2.slice/crio-eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8 WatchSource:0}: Error finding container eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8: Status 404 returned error can't find the container with id eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8 Nov 03 08:44:32.535863 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:32.535826 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerStarted","Data":"eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8"} Nov 03 08:44:35.052703 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:35.052664 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:35.565042 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:44:35.565000 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:44:36.456851 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:36.456827 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:44:36.457976 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:36.457953 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:44:37.348129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.348095 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7"] Nov 03 08:44:37.350907 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.350886 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.455362 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.455331 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x7xpb\"" Nov 03 08:44:37.456472 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.456448 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7"] Nov 03 08:44:37.525242 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.525205 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.525675 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.525273 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.525675 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.525400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2kt\" (UniqueName: \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.626006 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.625977 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2kt\" (UniqueName: \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.626214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.626065 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.626214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.626111 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.626556 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.626519 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.626675 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.626588 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.686325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.686286 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2kt\" (UniqueName: \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:37.964665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:37.964644 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:38.335412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.335392 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7"] Nov 03 08:44:38.337305 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:44:38.337278 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1154b3f_a089_4846_9693_46941cbaa67d.slice/crio-2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac WatchSource:0}: Error finding container 2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac: Status 404 returned error can't find the container with id 2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac Nov 03 08:44:38.559845 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.559812 4543 generic.go:356] "Generic (PLEG): container finished" podID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerID="778fd9c9428b5fb70b1ae430e820e96e8c66bdea9c096f90f02f594b34c60fdd" exitCode=0 Nov 03 08:44:38.560239 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.559899 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerDied","Data":"778fd9c9428b5fb70b1ae430e820e96e8c66bdea9c096f90f02f594b34c60fdd"} Nov 03 08:44:38.561048 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.561031 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:44:38.561599 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.561578 4543 generic.go:356] "Generic (PLEG): container finished" podID="b1154b3f-a089-4846-9693-46941cbaa67d" containerID="f4a1af389fcf2859881608409e0c450b57152d4a4158d78275c0fdef8b653c02" exitCode=0 Nov 03 08:44:38.561694 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.561642 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerDied","Data":"f4a1af389fcf2859881608409e0c450b57152d4a4158d78275c0fdef8b653c02"} Nov 03 08:44:38.561694 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:38.561683 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerStarted","Data":"2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac"} Nov 03 08:44:40.571873 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:40.571787 4543 generic.go:356] "Generic (PLEG): container finished" podID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerID="b6a107251a97de0e3dea2c2dc7989474b867d4eb0e56511fd92af5648cecb1ce" exitCode=0 Nov 03 08:44:40.571873 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:40.571835 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerDied","Data":"b6a107251a97de0e3dea2c2dc7989474b867d4eb0e56511fd92af5648cecb1ce"} Nov 03 08:44:42.585595 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:42.585559 4543 generic.go:356] "Generic (PLEG): container finished" podID="b1154b3f-a089-4846-9693-46941cbaa67d" containerID="58da31b7c2806cd76b4007ad73a75f49d0904669782bd598f60e80d17f921662" exitCode=0 Nov 03 08:44:42.586001 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:42.585641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerDied","Data":"58da31b7c2806cd76b4007ad73a75f49d0904669782bd598f60e80d17f921662"} Nov 03 08:44:47.608689 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:47.608651 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerStarted","Data":"d6a5ea1515e4e2df0964e79a81a551e6848970057c732fddbe24bdb6e53f1ce1"} Nov 03 08:44:47.729535 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:47.729439 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" podStartSLOduration=2.369405886 podStartE2EDuration="16.729424547s" totalImagesPullingTime="14.360018661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:47.728306006 +0000 UTC m=+311.396346746" watchObservedRunningTime="2026-11-03 08:44:47.729424547 +0000 UTC m=+311.397465286" Nov 03 08:44:48.614526 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:48.614487 4543 generic.go:356] "Generic (PLEG): container finished" podID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerID="d6a5ea1515e4e2df0964e79a81a551e6848970057c732fddbe24bdb6e53f1ce1" exitCode=0 Nov 03 08:44:48.615007 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:48.614576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerDied","Data":"d6a5ea1515e4e2df0964e79a81a551e6848970057c732fddbe24bdb6e53f1ce1"} Nov 03 08:44:49.620305 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.620262 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerStarted","Data":"54320074857d25c04f2abc158757be09a8dec87e09c96feea0cc36c81211fade"} Nov 03 08:44:49.739017 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.738957 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" podStartSLOduration=2.92116945 podStartE2EDuration="12.73894394s" totalImagesPullingTime="9.81777449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:49.73810576 +0000 UTC m=+313.406146502" watchObservedRunningTime="2026-11-03 08:44:49.73894394 +0000 UTC m=+313.406984678" Nov 03 08:44:49.742438 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.742416 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:49.829633 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.829597 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\" (UniqueName: \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\") pod \"da7b93cd-3326-48f2-8570-a467476bc6d2\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " Nov 03 08:44:49.829812 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.829646 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\") pod \"da7b93cd-3326-48f2-8570-a467476bc6d2\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " Nov 03 08:44:49.829812 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.829743 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\") pod \"da7b93cd-3326-48f2-8570-a467476bc6d2\" (UID: \"da7b93cd-3326-48f2-8570-a467476bc6d2\") " Nov 03 08:44:49.830131 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.830105 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle" pod "da7b93cd-3326-48f2-8570-a467476bc6d2" (UID: "da7b93cd-3326-48f2-8570-a467476bc6d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:44:49.831966 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.831944 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq" pod "da7b93cd-3326-48f2-8570-a467476bc6d2" (UID: "da7b93cd-3326-48f2-8570-a467476bc6d2"). InnerVolumeSpecName "kube-api-access-gmcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:44:49.834280 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.834258 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util" pod "da7b93cd-3326-48f2-8570-a467476bc6d2" (UID: "da7b93cd-3326-48f2-8570-a467476bc6d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:44:49.930276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.930243 4543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-bundle\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:49.930276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.930272 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcsq\" (UniqueName: \"kubernetes.io/projected/da7b93cd-3326-48f2-8570-a467476bc6d2-kube-api-access-gmcsq\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:49.930457 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:49.930283 4543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da7b93cd-3326-48f2-8570-a467476bc6d2-util\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:50.625041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:50.625007 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" event={"ID":"da7b93cd-3326-48f2-8570-a467476bc6d2","Type":"ContainerDied","Data":"eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8"} Nov 03 08:44:50.625041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:50.625035 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pkbg8" Nov 03 08:44:50.625583 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:50.625057 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb772862e11c01b5923a684765e1486615ac24d354bc670cd9fed391b62b31f8" Nov 03 08:44:50.627433 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:50.627406 4543 generic.go:356] "Generic (PLEG): container finished" podID="b1154b3f-a089-4846-9693-46941cbaa67d" containerID="54320074857d25c04f2abc158757be09a8dec87e09c96feea0cc36c81211fade" exitCode=0 Nov 03 08:44:50.627558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:50.627463 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerDied","Data":"54320074857d25c04f2abc158757be09a8dec87e09c96feea0cc36c81211fade"} Nov 03 08:44:51.764620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.764589 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:51.845808 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.845772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\" (UniqueName: \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\") pod \"b1154b3f-a089-4846-9693-46941cbaa67d\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " Nov 03 08:44:51.846008 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.845877 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\") pod \"b1154b3f-a089-4846-9693-46941cbaa67d\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " Nov 03 08:44:51.846008 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.845906 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\") pod \"b1154b3f-a089-4846-9693-46941cbaa67d\" (UID: \"b1154b3f-a089-4846-9693-46941cbaa67d\") " Nov 03 08:44:51.846356 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.846326 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle" pod "b1154b3f-a089-4846-9693-46941cbaa67d" (UID: "b1154b3f-a089-4846-9693-46941cbaa67d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:44:51.848073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.848048 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt" pod "b1154b3f-a089-4846-9693-46941cbaa67d" (UID: "b1154b3f-a089-4846-9693-46941cbaa67d"). InnerVolumeSpecName "kube-api-access-9h2kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:44:51.850891 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.850861 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util" pod "b1154b3f-a089-4846-9693-46941cbaa67d" (UID: "b1154b3f-a089-4846-9693-46941cbaa67d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:44:51.947026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.946945 4543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-bundle\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:51.947026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.946975 4543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b1154b3f-a089-4846-9693-46941cbaa67d-util\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:51.947026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:51.946985 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2kt\" (UniqueName: \"kubernetes.io/projected/b1154b3f-a089-4846-9693-46941cbaa67d-kube-api-access-9h2kt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:44:52.635912 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:52.635877 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" event={"ID":"b1154b3f-a089-4846-9693-46941cbaa67d","Type":"ContainerDied","Data":"2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac"} Nov 03 08:44:52.635912 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:52.635907 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644lj6v7" Nov 03 08:44:52.636109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:52.635927 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae4cedade1b8b322297a67925bbeefac461f0467580b044c50670ce09e020ac" Nov 03 08:44:57.611567 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:57.611535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:57.740888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:57.740855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:44:59.322683 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322652 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vp488"] Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322910 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="pull" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322921 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="pull" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322930 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="extract" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322934 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="extract" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322942 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="util" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322947 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="util" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322953 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="util" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322957 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="util" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322969 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="extract" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322974 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="extract" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322985 4543 cpu_manager.go:414] "containerMap: removing container" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="pull" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.322990 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="pull" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.323031 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7b93cd-3326-48f2-8570-a467476bc6d2" containerName="extract" Nov 03 08:44:59.323129 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.323040 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1154b3f-a089-4846-9693-46941cbaa67d" containerName="extract" Nov 03 08:44:59.365348 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.365323 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.390700 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.390679 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vp488"] Nov 03 08:44:59.402957 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.402926 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gh8\" (UniqueName: \"kubernetes.io/projected/d2d32aad-0569-4d35-a08c-44d13f35fe28-kube-api-access-w5gh8\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.403064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.402978 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d32aad-0569-4d35-a08c-44d13f35fe28-tmp\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.439993 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.439967 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-mjh89\"" Nov 03 08:44:59.440187 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.439998 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 08:44:59.440187 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.440043 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 08:44:59.503422 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.503387 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gh8\" (UniqueName: \"kubernetes.io/projected/d2d32aad-0569-4d35-a08c-44d13f35fe28-kube-api-access-w5gh8\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.503577 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.503551 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d32aad-0569-4d35-a08c-44d13f35fe28-tmp\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.503939 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.503922 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2d32aad-0569-4d35-a08c-44d13f35fe28-tmp\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.562697 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.562669 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gh8\" (UniqueName: \"kubernetes.io/projected/d2d32aad-0569-4d35-a08c-44d13f35fe28-kube-api-access-w5gh8\") pod \"jobset-operator-674cd59dc5-vp488\" (UID: \"d2d32aad-0569-4d35-a08c-44d13f35fe28\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:44:59.693291 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:44:59.693265 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" Nov 03 08:45:00.075566 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:00.075498 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-vp488"] Nov 03 08:45:00.078679 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:00.078646 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2d32aad_0569_4d35_a08c_44d13f35fe28.slice/crio-c4c0bcea379c8c863dbe94c8c50455a45357767e5018feb566e44b0e5752cc55 WatchSource:0}: Error finding container c4c0bcea379c8c863dbe94c8c50455a45357767e5018feb566e44b0e5752cc55: Status 404 returned error can't find the container with id c4c0bcea379c8c863dbe94c8c50455a45357767e5018feb566e44b0e5752cc55 Nov 03 08:45:00.663643 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:00.663611 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" event={"ID":"d2d32aad-0569-4d35-a08c-44d13f35fe28","Type":"ContainerStarted","Data":"c4c0bcea379c8c863dbe94c8c50455a45357767e5018feb566e44b0e5752cc55"} Nov 03 08:45:03.676333 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:03.676294 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" event={"ID":"d2d32aad-0569-4d35-a08c-44d13f35fe28","Type":"ContainerStarted","Data":"c4c31a8a8b7b35210d63dc4a3036bd9116a18c83880804092c591839ddff9ae4"} Nov 03 08:45:03.798405 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:03.798355 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-vp488" podStartSLOduration=1.7571501600000001 podStartE2EDuration="4.798339142s" totalImagesPullingTime="3.041188982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:03.797315728 +0000 UTC m=+327.465356489" watchObservedRunningTime="2026-11-03 08:45:03.798339142 +0000 UTC m=+327.466379881" Nov 03 08:45:05.559051 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:45:05.559017 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:45:05.871928 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:05.871899 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:18.799333 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:18.799303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:21.352181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:21.352106 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:23.318525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:23.318492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:23.463072 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:23.463039 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:24.711551 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:24.711515 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:24.970911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:24.970836 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv"] Nov 03 08:45:24.973118 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:24.973094 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.010725 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.010692 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:25.043464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.043438 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft"] Nov 03 08:45:25.045766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.045747 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:25.084461 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.084428 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv"] Nov 03 08:45:25.087197 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.087180 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 08:45:25.087803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.087788 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 08:45:25.088082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.088066 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 08:45:25.088162 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.088120 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 08:45:25.088226 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.088168 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tbsxf\"" Nov 03 08:45:25.088277 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.088232 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 08:45:25.099793 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.099771 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 08:45:25.099919 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.099812 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v25x6\"" Nov 03 08:45:25.100064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100043 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.100131 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100095 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.100215 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.100292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100276 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.100366 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100350 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.100409 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.100378 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnv6\" (UniqueName: \"kubernetes.io/projected/e676bda6-83ca-4fd4-920e-67fe53e41f16-kube-api-access-zwnv6\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.101179 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.101162 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 08:45:25.153689 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.153666 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft"] Nov 03 08:45:25.201244 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201215 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtnz\" (UniqueName: \"kubernetes.io/projected/209e8786-a2b0-454f-89b8-2b491daec379-kube-api-access-9mtnz\") pod \"limitador-operator-controller-manager-5778bb4449-nrqft\" (UID: \"209e8786-a2b0-454f-89b8-2b491daec379\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:25.201363 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201254 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201363 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201290 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201363 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201350 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201476 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201450 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201477 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201504 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnv6\" (UniqueName: \"kubernetes.io/projected/e676bda6-83ca-4fd4-920e-67fe53e41f16-kube-api-access-zwnv6\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.201856 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.201832 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.203877 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.203852 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e676bda6-83ca-4fd4-920e-67fe53e41f16-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.203985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.203924 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.204040 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.203981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.204236 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.204217 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e676bda6-83ca-4fd4-920e-67fe53e41f16-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.210066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.210043 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:25.264522 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.264472 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnv6\" (UniqueName: \"kubernetes.io/projected/e676bda6-83ca-4fd4-920e-67fe53e41f16-kube-api-access-zwnv6\") pod \"opendatahub-operator-controller-manager-77df88dcb4-4sfqv\" (UID: \"e676bda6-83ca-4fd4-920e-67fe53e41f16\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.296068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.296040 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:25.302084 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.302059 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtnz\" (UniqueName: \"kubernetes.io/projected/209e8786-a2b0-454f-89b8-2b491daec379-kube-api-access-9mtnz\") pod \"limitador-operator-controller-manager-5778bb4449-nrqft\" (UID: \"209e8786-a2b0-454f-89b8-2b491daec379\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:25.357057 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.357016 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:25.365966 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.365937 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtnz\" (UniqueName: \"kubernetes.io/projected/209e8786-a2b0-454f-89b8-2b491daec379-kube-api-access-9mtnz\") pod \"limitador-operator-controller-manager-5778bb4449-nrqft\" (UID: \"209e8786-a2b0-454f-89b8-2b491daec379\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:25.501159 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.501116 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:25.644675 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.644650 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:25.661735 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.661703 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:25.789397 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.789376 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv"] Nov 03 08:45:25.791643 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:25.791615 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode676bda6_83ca_4fd4_920e_67fe53e41f16.slice/crio-900353a7470debc71af31948996178fc7f388536b534bbac02f3ff4e98a83ec1 WatchSource:0}: Error finding container 900353a7470debc71af31948996178fc7f388536b534bbac02f3ff4e98a83ec1: Status 404 returned error can't find the container with id 900353a7470debc71af31948996178fc7f388536b534bbac02f3ff4e98a83ec1 Nov 03 08:45:25.794038 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:25.794014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:26.029800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:26.029763 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft"] Nov 03 08:45:26.032308 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:26.032284 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209e8786_a2b0_454f_89b8_2b491daec379.slice/crio-a18decec5b37a50093470c0200734595c5c3a04475407f1deb3d7dce288e70ee WatchSource:0}: Error finding container a18decec5b37a50093470c0200734595c5c3a04475407f1deb3d7dce288e70ee: Status 404 returned error can't find the container with id a18decec5b37a50093470c0200734595c5c3a04475407f1deb3d7dce288e70ee Nov 03 08:45:26.494988 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:26.494953 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:26.756226 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:26.756127 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" event={"ID":"209e8786-a2b0-454f-89b8-2b491daec379","Type":"ContainerStarted","Data":"a18decec5b37a50093470c0200734595c5c3a04475407f1deb3d7dce288e70ee"} Nov 03 08:45:26.757696 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:26.757658 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" event={"ID":"e676bda6-83ca-4fd4-920e-67fe53e41f16","Type":"ContainerStarted","Data":"900353a7470debc71af31948996178fc7f388536b534bbac02f3ff4e98a83ec1"} Nov 03 08:45:28.000037 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:27.999991 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:28.935973 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:28.935940 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:29.774726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:29.774689 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" event={"ID":"209e8786-a2b0-454f-89b8-2b491daec379","Type":"ContainerStarted","Data":"da6c4658eed57e27f5eee12772f8c04ccf81327ae3a7a1a875581237eb73a7ab"} Nov 03 08:45:29.775172 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:29.774782 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:29.895806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:29.895762 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" podStartSLOduration=2.267572716 podStartE2EDuration="5.895749274s" totalImagesPullingTime="3.628176558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:29.894862245 +0000 UTC m=+353.562903005" watchObservedRunningTime="2026-11-03 08:45:29.895749274 +0000 UTC m=+353.563790013" Nov 03 08:45:30.289012 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:30.288973 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:30.780050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:30.780015 4543 generic.go:356] "Generic (PLEG): container finished" podID="e676bda6-83ca-4fd4-920e-67fe53e41f16" containerID="0fec10f0c8eefb7b936b758c8b0671aa9020364e7a1e2b6e852b88827d496a44" exitCode=0 Nov 03 08:45:30.780555 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:30.780101 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" event={"ID":"e676bda6-83ca-4fd4-920e-67fe53e41f16","Type":"ContainerDied","Data":"0fec10f0c8eefb7b936b758c8b0671aa9020364e7a1e2b6e852b88827d496a44"} Nov 03 08:45:31.280942 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:31.280912 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:31.787785 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:31.787746 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" event={"ID":"e676bda6-83ca-4fd4-920e-67fe53e41f16","Type":"ContainerStarted","Data":"ecc98c4bca58489d5340e0c28e9cdfa203960bf6428a7d37e8e3fd8fefba6682"} Nov 03 08:45:31.788133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:31.787858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:31.972434 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:31.972342 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" podStartSLOduration=2.577025823 podStartE2EDuration="7.972329976s" totalImagesPullingTime="4.898724366s" totalInitContainerRuntime="496.579787ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:31.971198856 +0000 UTC m=+355.639239595" watchObservedRunningTime="2026-11-03 08:45:31.972329976 +0000 UTC m=+355.640370709" Nov 03 08:45:32.149476 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:32.149443 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:32.292436 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:32.292367 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:32.742929 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:32.742890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:34.073783 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:34.073755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:34.235348 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:34.235312 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:35.210904 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:35.210875 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:35.559810 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:45:35.559738 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:45:36.180486 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:36.180459 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:36.317514 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:36.317483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:38.695652 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:38.695623 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:39.136060 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:39.136022 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:40.782003 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:40.781973 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-nrqft" Nov 03 08:45:41.717799 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.717767 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr"] Nov 03 08:45:41.724544 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.724522 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:41.829810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.829782 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 08:45:41.830174 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.829828 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ghbpr\"" Nov 03 08:45:41.831026 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.831009 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wf7\" (UniqueName: \"kubernetes.io/projected/98880443-bd5a-416c-8e86-2eb9a2d8fb40-kube-api-access-f2wf7\") pod \"dns-operator-controller-manager-8557cbdb84-69hhr\" (UID: \"98880443-bd5a-416c-8e86-2eb9a2d8fb40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:41.831068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.831010 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr"] Nov 03 08:45:41.932030 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.932009 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wf7\" (UniqueName: \"kubernetes.io/projected/98880443-bd5a-416c-8e86-2eb9a2d8fb40-kube-api-access-f2wf7\") pod \"dns-operator-controller-manager-8557cbdb84-69hhr\" (UID: \"98880443-bd5a-416c-8e86-2eb9a2d8fb40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:41.994119 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:41.994058 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wf7\" (UniqueName: \"kubernetes.io/projected/98880443-bd5a-416c-8e86-2eb9a2d8fb40-kube-api-access-f2wf7\") pod \"dns-operator-controller-manager-8557cbdb84-69hhr\" (UID: \"98880443-bd5a-416c-8e86-2eb9a2d8fb40\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:42.036027 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:42.036002 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:42.411166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:42.409099 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr"] Nov 03 08:45:42.436475 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:42.436440 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98880443_bd5a_416c_8e86_2eb9a2d8fb40.slice/crio-466b81eb391ee9a0b68ddcdaf65cd8be856c014c51038aaf423535f0b7822199 WatchSource:0}: Error finding container 466b81eb391ee9a0b68ddcdaf65cd8be856c014c51038aaf423535f0b7822199: Status 404 returned error can't find the container with id 466b81eb391ee9a0b68ddcdaf65cd8be856c014c51038aaf423535f0b7822199 Nov 03 08:45:42.793850 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:42.793771 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-4sfqv" Nov 03 08:45:42.831105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:42.831071 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" event={"ID":"98880443-bd5a-416c-8e86-2eb9a2d8fb40","Type":"ContainerStarted","Data":"466b81eb391ee9a0b68ddcdaf65cd8be856c014c51038aaf423535f0b7822199"} Nov 03 08:45:43.699171 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:43.699123 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:43.834915 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:43.834881 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:45.843853 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:45.843824 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" event={"ID":"98880443-bd5a-416c-8e86-2eb9a2d8fb40","Type":"ContainerStarted","Data":"cb1b5a5f4ed87f293e0c6f07568bd22cd1ffbce30d44657a7819531e4f7e07fc"} Nov 03 08:45:45.844256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:45.843899 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:45.963600 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:45.963530 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" podStartSLOduration=2.553316618 podStartE2EDuration="4.963508023s" totalImagesPullingTime="2.410191405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:45.962231023 +0000 UTC m=+369.630271763" watchObservedRunningTime="2026-11-03 08:45:45.963508023 +0000 UTC m=+369.631548772" Nov 03 08:45:48.303629 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:48.303594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:48.505603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:48.505574 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:48.896419 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:48.896390 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:49.027164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:49.027108 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:50.738216 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.738179 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:50.779355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.779329 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4"] Nov 03 08:45:50.782587 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.782567 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.804049 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.804028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/a4205e96-340e-40e7-8071-24d8206086ea-kube-api-access-s8rgq\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.804164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.804066 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4205e96-340e-40e7-8071-24d8206086ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.804164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.804104 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4205e96-340e-40e7-8071-24d8206086ea-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.840504 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.840484 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4"] Nov 03 08:45:50.889628 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.889609 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 08:45:50.889628 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.889624 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 08:45:50.889758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.889631 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-znzl4\"" Nov 03 08:45:50.905273 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.905249 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4205e96-340e-40e7-8071-24d8206086ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.905368 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.905287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4205e96-340e-40e7-8071-24d8206086ea-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.905425 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.905362 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/a4205e96-340e-40e7-8071-24d8206086ea-kube-api-access-s8rgq\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.905950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.905928 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a4205e96-340e-40e7-8071-24d8206086ea-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.907729 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.907710 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a4205e96-340e-40e7-8071-24d8206086ea-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:50.961480 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.961460 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:50.964762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:50.964740 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rgq\" (UniqueName: \"kubernetes.io/projected/a4205e96-340e-40e7-8071-24d8206086ea-kube-api-access-s8rgq\") pod \"kuadrant-console-plugin-66c4fb575d-5w7h4\" (UID: \"a4205e96-340e-40e7-8071-24d8206086ea\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:51.093655 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:51.093607 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" Nov 03 08:45:51.460720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:51.460607 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4"] Nov 03 08:45:51.463326 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:51.463298 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4205e96_340e_40e7_8071_24d8206086ea.slice/crio-66f73a038eba4f31be1c3995694245d67b8d6b4298c8c99f2b9a718b343c872e WatchSource:0}: Error finding container 66f73a038eba4f31be1c3995694245d67b8d6b4298c8c99f2b9a718b343c872e: Status 404 returned error can't find the container with id 66f73a038eba4f31be1c3995694245d67b8d6b4298c8c99f2b9a718b343c872e Nov 03 08:45:51.869345 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:51.869313 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" event={"ID":"a4205e96-340e-40e7-8071-24d8206086ea","Type":"ContainerStarted","Data":"66f73a038eba4f31be1c3995694245d67b8d6b4298c8c99f2b9a718b343c872e"} Nov 03 08:45:52.001393 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:52.001362 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:52.222766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:52.222698 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:53.524814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:53.524780 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:53.661155 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:53.661097 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:53.794454 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:53.794373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:54.243382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:54.243342 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:54.407797 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:54.407764 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:54.553735 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:54.553660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:54.867348 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:54.867315 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:55.010879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:55.010834 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:56.197432 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.197398 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:56.551874 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.551801 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:56.683045 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.683015 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:56.821496 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.821414 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:56.844751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.844721 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg"] Nov 03 08:45:56.851261 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.851239 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-69hhr" Nov 03 08:45:56.851405 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.851338 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:56.909531 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.909499 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg"] Nov 03 08:45:56.910262 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.910239 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 08:45:56.910262 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.910248 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 08:45:56.910418 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.910336 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w8mcg\"" Nov 03 08:45:56.911091 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.911071 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 08:45:56.911239 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.911185 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 08:45:56.960515 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.960480 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78fr2\" (UniqueName: \"kubernetes.io/projected/2df8810e-ad76-452e-9da1-581d58a9b9b7-kube-api-access-78fr2\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:56.960687 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.960659 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df8810e-ad76-452e-9da1-581d58a9b9b7-tls-certs\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:56.960800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.960781 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2df8810e-ad76-452e-9da1-581d58a9b9b7-tmp\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:56.969384 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:56.969326 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:57.062015 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.061985 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2df8810e-ad76-452e-9da1-581d58a9b9b7-tmp\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.062196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.062027 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78fr2\" (UniqueName: \"kubernetes.io/projected/2df8810e-ad76-452e-9da1-581d58a9b9b7-kube-api-access-78fr2\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.062196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.062078 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df8810e-ad76-452e-9da1-581d58a9b9b7-tls-certs\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.064357 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.064326 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2df8810e-ad76-452e-9da1-581d58a9b9b7-tmp\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.064518 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.064501 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df8810e-ad76-452e-9da1-581d58a9b9b7-tls-certs\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.121634 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.121609 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78fr2\" (UniqueName: \"kubernetes.io/projected/2df8810e-ad76-452e-9da1-581d58a9b9b7-kube-api-access-78fr2\") pod \"kube-auth-proxy-677799c6bb-6w6tg\" (UID: \"2df8810e-ad76-452e-9da1-581d58a9b9b7\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.167596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.167569 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" Nov 03 08:45:57.513629 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.513564 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:57.539457 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.539440 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg"] Nov 03 08:45:57.541721 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:57.541692 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df8810e_ad76_452e_9da1_581d58a9b9b7.slice/crio-c3b79d455120b41e864c33cbe1cb20669906b41d3e77a81e20c3dd00b56de9e2 WatchSource:0}: Error finding container c3b79d455120b41e864c33cbe1cb20669906b41d3e77a81e20c3dd00b56de9e2: Status 404 returned error can't find the container with id c3b79d455120b41e864c33cbe1cb20669906b41d3e77a81e20c3dd00b56de9e2 Nov 03 08:45:57.660188 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.660161 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:57.907769 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.907733 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" event={"ID":"2df8810e-ad76-452e-9da1-581d58a9b9b7","Type":"ContainerStarted","Data":"c3b79d455120b41e864c33cbe1cb20669906b41d3e77a81e20c3dd00b56de9e2"} Nov 03 08:45:57.909685 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:57.909656 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" event={"ID":"a4205e96-340e-40e7-8071-24d8206086ea","Type":"ContainerStarted","Data":"f35597624055a2c5dff72db9e71f74ccecc251511e292d12bf88c634b3baf2e8"} Nov 03 08:45:58.026178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.026030 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-5w7h4" podStartSLOduration=2.259226947 podStartE2EDuration="8.026012816s" totalImagesPullingTime="5.766785869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:58.025785873 +0000 UTC m=+381.693826611" watchObservedRunningTime="2026-11-03 08:45:58.026012816 +0000 UTC m=+381.694053556" Nov 03 08:45:58.187094 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.187015 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:58.372287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.372259 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:45:58.612225 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.612201 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4"] Nov 03 08:45:58.615965 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.615944 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.673474 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.673449 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 08:45:58.674446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.674420 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4"] Nov 03 08:45:58.675047 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.675027 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 08:45:58.676343 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.676320 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmxs\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-kube-api-access-2mmxs\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.676431 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.676361 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.676618 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.676599 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xvkjp\"" Nov 03 08:45:58.776871 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.776840 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmxs\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-kube-api-access-2mmxs\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.777051 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.776886 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.849857 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.849825 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.851106 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.851078 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmxs\" (UniqueName: \"kubernetes.io/projected/d53304b8-5e14-4e3a-82b9-c4d4799ddd09-kube-api-access-2mmxs\") pod \"cert-manager-cainjector-5c57bbd4fc-nm7k4\" (UID: \"d53304b8-5e14-4e3a-82b9-c4d4799ddd09\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:58.932202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:58.932117 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" Nov 03 08:45:59.320353 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:59.320291 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4"] Nov 03 08:45:59.323597 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:45:59.323568 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53304b8_5e14_4e3a_82b9_c4d4799ddd09.slice/crio-e4ec208a90b294bc2fb3652f9fa27f2851a4196b2222fa42651f6ed26a5bfff5 WatchSource:0}: Error finding container e4ec208a90b294bc2fb3652f9fa27f2851a4196b2222fa42651f6ed26a5bfff5: Status 404 returned error can't find the container with id e4ec208a90b294bc2fb3652f9fa27f2851a4196b2222fa42651f6ed26a5bfff5 Nov 03 08:45:59.921888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:45:59.921829 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" event={"ID":"d53304b8-5e14-4e3a-82b9-c4d4799ddd09","Type":"ContainerStarted","Data":"e4ec208a90b294bc2fb3652f9fa27f2851a4196b2222fa42651f6ed26a5bfff5"} Nov 03 08:46:00.023341 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.023303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:00.157411 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.157378 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:00.795229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.795151 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:46:00.798645 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.798619 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:46:00.854243 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.854215 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:46:00.895374 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.895345 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xh9d\" (UniqueName: \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\") pod \"authorino-56c6c9ddd5-2b8cr\" (UID: \"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618\") " pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:46:00.903244 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.903219 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-h722q\"" Nov 03 08:46:00.928507 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.928472 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" event={"ID":"2df8810e-ad76-452e-9da1-581d58a9b9b7","Type":"ContainerStarted","Data":"8cdd7f5fdfc067cd8e0e14bb43ef2423a767c5fc5e6ec1a5fad17f227dbad4ca"} Nov 03 08:46:00.996113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:00.996080 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xh9d\" (UniqueName: \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\") pod \"authorino-56c6c9ddd5-2b8cr\" (UID: \"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618\") " pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:46:01.048813 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:01.048727 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-677799c6bb-6w6tg" podStartSLOduration=2.185144096 podStartE2EDuration="5.048712178s" totalImagesPullingTime="2.863568082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:01.047177005 +0000 UTC m=+384.715217746" watchObservedRunningTime="2026-11-03 08:46:01.048712178 +0000 UTC m=+384.716752937" Nov 03 08:46:01.056800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:01.056773 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xh9d\" (UniqueName: \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\") pod \"authorino-56c6c9ddd5-2b8cr\" (UID: \"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618\") " pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:46:01.112788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:01.112747 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:46:01.499252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:01.498935 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:46:01.501700 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:46:01.501663 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8e8eb18_1ab5_4bb7_8e8a_5393c9dae618.slice/crio-5dc99ce8530b1f7918fdb3be545fdd2378c4d61c091e11f22a517675ca5d23ef WatchSource:0}: Error finding container 5dc99ce8530b1f7918fdb3be545fdd2378c4d61c091e11f22a517675ca5d23ef: Status 404 returned error can't find the container with id 5dc99ce8530b1f7918fdb3be545fdd2378c4d61c091e11f22a517675ca5d23ef Nov 03 08:46:01.934918 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:01.934885 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" event={"ID":"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618","Type":"ContainerStarted","Data":"5dc99ce8530b1f7918fdb3be545fdd2378c4d61c091e11f22a517675ca5d23ef"} Nov 03 08:46:02.969825 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:02.969785 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:03.945585 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:03.945551 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" event={"ID":"d53304b8-5e14-4e3a-82b9-c4d4799ddd09","Type":"ContainerStarted","Data":"843ae83040b122fa81df2c8bd054a25bad8d9d1ef68e0b2865615e0e0159f1f1"} Nov 03 08:46:03.947198 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:03.947170 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" event={"ID":"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618","Type":"ContainerStarted","Data":"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88"} Nov 03 08:46:04.064293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:04.064173 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-nm7k4" podStartSLOduration=2.009435327 podStartE2EDuration="6.064153956s" totalImagesPullingTime="4.054718629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:04.062304447 +0000 UTC m=+387.730345186" watchObservedRunningTime="2026-11-03 08:46:04.064153956 +0000 UTC m=+387.732194694" Nov 03 08:46:04.179694 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:04.179643 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" podStartSLOduration=1.885162923 podStartE2EDuration="4.179629939s" totalImagesPullingTime="2.294467016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:04.17800589 +0000 UTC m=+387.846046630" watchObservedRunningTime="2026-11-03 08:46:04.179629939 +0000 UTC m=+387.847670678" Nov 03 08:46:04.435317 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:04.435288 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:05.035054 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:05.035024 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:05.201270 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:05.201240 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:05.560780 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:46:05.560742 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:46:05.963301 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:05.962409 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:06.093536 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:06.093502 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:06.592448 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:06.592410 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:06.723306 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:06.723279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:11.201759 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:11.201729 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:11.341839 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:11.341809 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:11.801440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:11.801408 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:11.935477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:11.935446 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:12.939334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:12.939291 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hhhdr"] Nov 03 08:46:12.944091 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:12.944064 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:12.994455 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:12.994428 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-bound-sa-token\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:12.994600 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:12.994478 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tclsq\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-kube-api-access-tclsq\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.001470 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.001446 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hhhdr"] Nov 03 08:46:13.050071 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.050048 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bwcdg\"" Nov 03 08:46:13.095492 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.095466 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-bound-sa-token\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.095600 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.095514 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tclsq\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-kube-api-access-tclsq\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.156170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.156129 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-bound-sa-token\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.156332 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.156315 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tclsq\" (UniqueName: \"kubernetes.io/projected/47ce01da-7ee4-4fe5-af89-f5953af91c65-kube-api-access-tclsq\") pod \"cert-manager-597b95cb84-hhhdr\" (UID: \"47ce01da-7ee4-4fe5-af89-f5953af91c65\") " pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.257089 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.257017 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hhhdr" Nov 03 08:46:13.628945 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.625987 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hhhdr"] Nov 03 08:46:13.631903 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:46:13.631879 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47ce01da_7ee4_4fe5_af89_f5953af91c65.slice/crio-fb8fd8fa12fcfb7a8a66be5aec6d371f767bc65af72bfe9caf4ee4832c3e7477 WatchSource:0}: Error finding container fb8fd8fa12fcfb7a8a66be5aec6d371f767bc65af72bfe9caf4ee4832c3e7477: Status 404 returned error can't find the container with id fb8fd8fa12fcfb7a8a66be5aec6d371f767bc65af72bfe9caf4ee4832c3e7477 Nov 03 08:46:13.989814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.989733 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hhhdr" event={"ID":"47ce01da-7ee4-4fe5-af89-f5953af91c65","Type":"ContainerStarted","Data":"6542304f8aa03a9e91b625d16b21c80a6feb28d66df16066bf9a85e75f618daa"} Nov 03 08:46:13.989814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:13.989781 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hhhdr" event={"ID":"47ce01da-7ee4-4fe5-af89-f5953af91c65","Type":"ContainerStarted","Data":"fb8fd8fa12fcfb7a8a66be5aec6d371f767bc65af72bfe9caf4ee4832c3e7477"} Nov 03 08:46:14.106870 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:14.106820 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hhhdr" podStartSLOduration=2.106807615 podStartE2EDuration="2.106807615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:14.105878494 +0000 UTC m=+397.773919429" watchObservedRunningTime="2026-11-03 08:46:14.106807615 +0000 UTC m=+397.774848352" Nov 03 08:46:17.206824 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:17.206756 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:17.347025 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:17.346994 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:17.750085 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:17.750049 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:17.881455 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:17.881409 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:18.396694 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:18.396623 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:18.531616 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:18.531570 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:19.149288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:19.149251 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:21.717522 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:21.717473 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:21.864574 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:21.864532 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:22.482972 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:22.482939 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:25.101822 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:25.101792 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:25.231761 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:25.231732 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:32.239451 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:32.239412 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:32.382494 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:32.382449 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:32.514834 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:32.514755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:35.559984 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:46:35.559949 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:46:52.268053 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.267997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:52.347440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.347407 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:46:52.350699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.350667 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.406511 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.406486 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 08:46:52.406691 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.406675 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 08:46:52.414463 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.414434 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.414571 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.414506 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.414571 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.414541 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z77h\" (UniqueName: \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.420731 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.420696 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:52.457166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.457123 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 08:46:52.457300 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.457272 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k54gs\"" Nov 03 08:46:52.458775 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.458758 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:46:52.506707 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.506687 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 08:46:52.506707 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.506687 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 08:46:52.515695 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.515668 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z77h\" (UniqueName: \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.515770 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.515730 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.515805 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.515782 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.518293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.518238 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.518293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.518259 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.560536 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.560515 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:52.575316 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.575290 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z77h\" (UniqueName: \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\") pod \"maas-controller-ff55dbc89-qtxxz\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.663307 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.663283 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:46:52.694356 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:52.694325 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:53.033380 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:53.033347 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:46:53.068197 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:46:53.068167 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b185e4_bf53_4b63_924d_74ebfda26542.slice/crio-7eadc6e95d0568dce008492e19dbaac1b3d78016411c7bc86d0eb0968cbfec54 WatchSource:0}: Error finding container 7eadc6e95d0568dce008492e19dbaac1b3d78016411c7bc86d0eb0968cbfec54: Status 404 returned error can't find the container with id 7eadc6e95d0568dce008492e19dbaac1b3d78016411c7bc86d0eb0968cbfec54 Nov 03 08:46:53.156412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:53.156378 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" event={"ID":"83b185e4-bf53-4b63-924d-74ebfda26542","Type":"ContainerStarted","Data":"7eadc6e95d0568dce008492e19dbaac1b3d78016411c7bc86d0eb0968cbfec54"} Nov 03 08:46:59.719325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:59.719291 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:46:59.855116 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:46:59.855082 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:00.189944 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:00.189912 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" event={"ID":"83b185e4-bf53-4b63-924d-74ebfda26542","Type":"ContainerStarted","Data":"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c"} Nov 03 08:47:00.190180 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:00.190163 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:47:03.054373 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:03.054342 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:03.380405 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:03.380369 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:05.572545 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:47:05.572503 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:47:10.815979 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:10.815923 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" podStartSLOduration=12.541322551 podStartE2EDuration="18.815909636s" totalImagesPullingTime="6.274587085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:00.368501366 +0000 UTC m=+444.036542103" watchObservedRunningTime="2026-11-03 08:47:10.815909636 +0000 UTC m=+454.483950373" Nov 03 08:47:10.817131 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:10.817113 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:47:10.817339 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:10.817317 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" podUID="83b185e4-bf53-4b63-924d-74ebfda26542" containerName="manager" containerID="cri-o://939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c" gracePeriod=30 Nov 03 08:47:10.819237 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:10.819216 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:47:11.155787 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.155766 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:47:11.238098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.238071 4543 generic.go:356] "Generic (PLEG): container finished" podID="83b185e4-bf53-4b63-924d-74ebfda26542" containerID="939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c" exitCode=0 Nov 03 08:47:11.238254 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.238109 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" Nov 03 08:47:11.238254 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.238172 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" event={"ID":"83b185e4-bf53-4b63-924d-74ebfda26542","Type":"ContainerDied","Data":"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c"} Nov 03 08:47:11.238254 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.238220 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-ff55dbc89-qtxxz" event={"ID":"83b185e4-bf53-4b63-924d-74ebfda26542","Type":"ContainerDied","Data":"7eadc6e95d0568dce008492e19dbaac1b3d78016411c7bc86d0eb0968cbfec54"} Nov 03 08:47:11.238254 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.238242 4543 scope.go:122] "RemoveContainer" containerID="939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c" Nov 03 08:47:11.245929 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.245906 4543 scope.go:122] "RemoveContainer" containerID="939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c" Nov 03 08:47:11.246201 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:47:11.246182 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c\": container with ID starting with 939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c not found: ID does not exist" containerID="939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c" Nov 03 08:47:11.246272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.246214 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c"} err="failed to get container status \"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c\": rpc error: code = NotFound desc = could not find container \"939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c\": container with ID starting with 939e5996f6bb72d7451054c70a46ce13e96ef646a34e979899bee57f381dbb4c not found: ID does not exist" Nov 03 08:47:11.265314 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.265297 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\") pod \"83b185e4-bf53-4b63-924d-74ebfda26542\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " Nov 03 08:47:11.265397 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.265334 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\" (UniqueName: \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\") pod \"83b185e4-bf53-4b63-924d-74ebfda26542\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " Nov 03 08:47:11.265397 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.265377 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\") pod \"83b185e4-bf53-4b63-924d-74ebfda26542\" (UID: \"83b185e4-bf53-4b63-924d-74ebfda26542\") " Nov 03 08:47:11.267503 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.267482 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h" pod "83b185e4-bf53-4b63-924d-74ebfda26542" (UID: "83b185e4-bf53-4b63-924d-74ebfda26542"). InnerVolumeSpecName "kube-api-access-5z77h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:47:11.267503 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.267487 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs" pod "83b185e4-bf53-4b63-924d-74ebfda26542" (UID: "83b185e4-bf53-4b63-924d-74ebfda26542"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:47:11.267625 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.267517 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert" pod "83b185e4-bf53-4b63-924d-74ebfda26542" (UID: "83b185e4-bf53-4b63-924d-74ebfda26542"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:47:11.366371 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.366350 4543 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-webhook-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:11.366371 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.366372 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83b185e4-bf53-4b63-924d-74ebfda26542-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:11.366519 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.366382 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z77h\" (UniqueName: \"kubernetes.io/projected/83b185e4-bf53-4b63-924d-74ebfda26542-kube-api-access-5z77h\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:11.660593 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.660526 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:47:11.711059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:11.711033 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-ff55dbc89-qtxxz"] Nov 03 08:47:12.449795 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:12.449767 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b185e4-bf53-4b63-924d-74ebfda26542" path="/var/lib/kubelet/pods/83b185e4-bf53-4b63-924d-74ebfda26542/volumes" Nov 03 08:47:16.978812 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:16.978778 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.219233 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.219183 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.377315 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.377281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.508892 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.508857 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.642359 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.642282 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.774267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.774238 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.909175 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.909009 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:17.910447 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.910426 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n64sg"] Nov 03 08:47:17.910731 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.910721 4543 cpu_manager.go:414] "containerMap: removing container" podUID="83b185e4-bf53-4b63-924d-74ebfda26542" containerName="manager" Nov 03 08:47:17.910774 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.910733 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b185e4-bf53-4b63-924d-74ebfda26542" containerName="manager" Nov 03 08:47:17.910818 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.910809 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b185e4-bf53-4b63-924d-74ebfda26542" containerName="manager" Nov 03 08:47:17.913067 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.913053 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:17.968133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.968106 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 08:47:17.968133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:17.968123 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 08:47:18.017437 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.017407 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p6648\"" Nov 03 08:47:18.017850 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.017543 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 08:47:18.017850 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.017579 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0124bfb9-f59d-47cf-904b-041308f81dca-cert\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.017850 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.017615 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskfs\" (UniqueName: \"kubernetes.io/projected/0124bfb9-f59d-47cf-904b-041308f81dca-kube-api-access-tskfs\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.019901 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.019880 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n64sg"] Nov 03 08:47:18.049018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.048993 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:18.118965 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.118935 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0124bfb9-f59d-47cf-904b-041308f81dca-cert\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.119104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.118970 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskfs\" (UniqueName: \"kubernetes.io/projected/0124bfb9-f59d-47cf-904b-041308f81dca-kube-api-access-tskfs\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.121482 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.121461 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0124bfb9-f59d-47cf-904b-041308f81dca-cert\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.146784 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.146759 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4s8bk"] Nov 03 08:47:18.148836 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.148819 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.178929 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.178883 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskfs\" (UniqueName: \"kubernetes.io/projected/0124bfb9-f59d-47cf-904b-041308f81dca-kube-api-access-tskfs\") pod \"kserve-controller-manager-776cc66d47-n64sg\" (UID: \"0124bfb9-f59d-47cf-904b-041308f81dca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.183199 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.183177 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:18.203524 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.203501 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-v8nmp\"" Nov 03 08:47:18.203524 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.203503 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 08:47:18.208915 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.208897 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4s8bk"] Nov 03 08:47:18.219678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.219654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdhn\" (UniqueName: \"kubernetes.io/projected/bec5a14a-dcb9-4022-8849-2b4553225759-kube-api-access-jfdhn\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.219772 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.219740 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec5a14a-dcb9-4022-8849-2b4553225759-tls-certs\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.223609 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.223590 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:18.278956 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.278929 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-wg2dn"] Nov 03 08:47:18.285481 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.285454 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.320413 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.320392 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdhn\" (UniqueName: \"kubernetes.io/projected/bec5a14a-dcb9-4022-8849-2b4553225759-kube-api-access-jfdhn\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.320554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.320445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec5a14a-dcb9-4022-8849-2b4553225759-tls-certs\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.322745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.322722 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec5a14a-dcb9-4022-8849-2b4553225759-tls-certs\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.342257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.342231 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 08:47:18.342257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.342233 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 08:47:18.342389 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.342289 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8hwrb\"" Nov 03 08:47:18.381333 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.381309 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdhn\" (UniqueName: \"kubernetes.io/projected/bec5a14a-dcb9-4022-8849-2b4553225759-kube-api-access-jfdhn\") pod \"model-serving-api-75bd89fdfc-4s8bk\" (UID: \"bec5a14a-dcb9-4022-8849-2b4553225759\") " pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.383029 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.383006 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wg2dn"] Nov 03 08:47:18.421263 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.421238 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtdw\" (UniqueName: \"kubernetes.io/projected/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-kube-api-access-xhtdw\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.421362 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.421271 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-cert\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.421400 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.421372 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-metrics-certs\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.459828 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.459774 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:18.522701 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.522674 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-metrics-certs\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.522811 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.522738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtdw\" (UniqueName: \"kubernetes.io/projected/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-kube-api-access-xhtdw\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.522811 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.522776 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-cert\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.524998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.524971 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-metrics-certs\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.525111 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.525087 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-cert\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.584366 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.584337 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtdw\" (UniqueName: \"kubernetes.io/projected/1cac33c5-bd20-499b-b8ec-78aee6e9faf8-kube-api-access-xhtdw\") pod \"odh-model-controller-769875db56-wg2dn\" (UID: \"1cac33c5-bd20-499b-b8ec-78aee6e9faf8\") " pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.593781 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.593758 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n64sg"] Nov 03 08:47:18.595450 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:47:18.595430 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0124bfb9_f59d_47cf_904b_041308f81dca.slice/crio-2e4838bf3c0507f1c5e349203a140121d9521bc5d925a534821d6fc5aa665d41 WatchSource:0}: Error finding container 2e4838bf3c0507f1c5e349203a140121d9521bc5d925a534821d6fc5aa665d41: Status 404 returned error can't find the container with id 2e4838bf3c0507f1c5e349203a140121d9521bc5d925a534821d6fc5aa665d41 Nov 03 08:47:18.602752 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.602708 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:18.827755 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.827731 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-4s8bk"] Nov 03 08:47:18.829619 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:47:18.829593 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec5a14a_dcb9_4022_8849_2b4553225759.slice/crio-955729e7a5c7d056332362c73b48801d3757a5c5bd583014b7c2e32d4086f21c WatchSource:0}: Error finding container 955729e7a5c7d056332362c73b48801d3757a5c5bd583014b7c2e32d4086f21c: Status 404 returned error can't find the container with id 955729e7a5c7d056332362c73b48801d3757a5c5bd583014b7c2e32d4086f21c Nov 03 08:47:18.968198 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:18.968178 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wg2dn"] Nov 03 08:47:18.970272 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:47:18.970248 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cac33c5_bd20_499b_b8ec_78aee6e9faf8.slice/crio-ad2084170098c3a2d40ee7d5903e1268cf2255a817f41b435f588e827663baad WatchSource:0}: Error finding container ad2084170098c3a2d40ee7d5903e1268cf2255a817f41b435f588e827663baad: Status 404 returned error can't find the container with id ad2084170098c3a2d40ee7d5903e1268cf2255a817f41b435f588e827663baad Nov 03 08:47:19.272604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:19.272569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" event={"ID":"bec5a14a-dcb9-4022-8849-2b4553225759","Type":"ContainerStarted","Data":"955729e7a5c7d056332362c73b48801d3757a5c5bd583014b7c2e32d4086f21c"} Nov 03 08:47:19.273799 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:19.273767 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wg2dn" event={"ID":"1cac33c5-bd20-499b-b8ec-78aee6e9faf8","Type":"ContainerStarted","Data":"ad2084170098c3a2d40ee7d5903e1268cf2255a817f41b435f588e827663baad"} Nov 03 08:47:19.274934 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:19.274911 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" event={"ID":"0124bfb9-f59d-47cf-904b-041308f81dca","Type":"ContainerStarted","Data":"2e4838bf3c0507f1c5e349203a140121d9521bc5d925a534821d6fc5aa665d41"} Nov 03 08:47:19.304884 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:19.304860 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:19.478010 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:19.477980 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:22.299335 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.299297 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" event={"ID":"bec5a14a-dcb9-4022-8849-2b4553225759","Type":"ContainerStarted","Data":"2a8e5f7932405779601003b1d2d188eb0738c2897c2a6e1051ba1f05d868da7a"} Nov 03 08:47:22.299813 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.299369 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:22.301068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.301040 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" event={"ID":"0124bfb9-f59d-47cf-904b-041308f81dca","Type":"ContainerStarted","Data":"39ac79417474a0bd814854f03fe29eec45accfa29c169d0504c4cc5e7831c901"} Nov 03 08:47:22.301227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.301167 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:22.419446 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.419407 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" podStartSLOduration=1.814033316 podStartE2EDuration="4.419392938s" totalImagesPullingTime="2.605359622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:22.418351194 +0000 UTC m=+466.086391936" watchObservedRunningTime="2026-11-03 08:47:22.419392938 +0000 UTC m=+466.087433675" Nov 03 08:47:22.535861 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:22.535809 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" podStartSLOduration=2.706533804 podStartE2EDuration="5.535792174s" totalImagesPullingTime="2.82925837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:22.534555611 +0000 UTC m=+466.202596350" watchObservedRunningTime="2026-11-03 08:47:22.535792174 +0000 UTC m=+466.203832913" Nov 03 08:47:23.307355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:23.307314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wg2dn" event={"ID":"1cac33c5-bd20-499b-b8ec-78aee6e9faf8","Type":"ContainerStarted","Data":"876f8bc34730261bd0ba463649aac64a263700a181b94736c424f870af989fc9"} Nov 03 08:47:23.307794 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:23.307556 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:23.424096 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:23.423751 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-wg2dn" podStartSLOduration=2.040474414 podStartE2EDuration="5.423735885s" totalImagesPullingTime="3.383261471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:23.423490015 +0000 UTC m=+467.091530753" watchObservedRunningTime="2026-11-03 08:47:23.423735885 +0000 UTC m=+467.091776625" Nov 03 08:47:24.521791 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:24.521762 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:24.651598 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:24.651561 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:24.775891 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:24.775823 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:24.906343 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:24.906310 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.042494 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.042419 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.186960 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.186926 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.313382 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.313295 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.444960 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.444933 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.766614 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.766581 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:25.900560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:25.900527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:29.900820 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:29.900787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:30.093562 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:30.093529 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:30.324194 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:30.324103 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:30.466814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:30.466784 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:32.516817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:32.516780 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:32.685279 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:32.685249 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:32.812808 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:32.812737 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:32.939843 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:32.939816 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:33.311613 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.311593 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-4s8bk" Nov 03 08:47:33.576902 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.576833 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:33.584080 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.584063 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.638882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.638852 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:33.639323 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.639304 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 08:47:33.639409 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.639342 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 08:47:33.639409 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.639384 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 08:47:33.651640 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.651621 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.651753 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.651646 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhds\" (UniqueName: \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.651753 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.651688 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.651862 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.651833 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.687053 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.687033 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 08:47:33.687197 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.687033 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-f42g7\"" Nov 03 08:47:33.752706 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.752684 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.752803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.752725 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.752803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.752754 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.752888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.752867 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhds\" (UniqueName: \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.753539 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.753458 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.755293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.755273 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.755380 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.755282 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.820898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.820868 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhds\" (UniqueName: \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\") pod \"maas-api-67bdd9df7-89nkl\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:33.896627 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:33.896599 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:34.266661 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:34.266638 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:34.268369 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:47:34.268344 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7451df_e714_4f63_a696_c3078b0f7818.slice/crio-a06d97d47a53af806547394becdb88d361530016fbe7f2d9e206e4616f4dc888 WatchSource:0}: Error finding container a06d97d47a53af806547394becdb88d361530016fbe7f2d9e206e4616f4dc888: Status 404 returned error can't find the container with id a06d97d47a53af806547394becdb88d361530016fbe7f2d9e206e4616f4dc888 Nov 03 08:47:34.314232 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:34.314211 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-wg2dn" Nov 03 08:47:34.358252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:34.358225 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" event={"ID":"bc7451df-e714-4f63-a696-c3078b0f7818","Type":"ContainerStarted","Data":"a06d97d47a53af806547394becdb88d361530016fbe7f2d9e206e4616f4dc888"} Nov 03 08:47:35.559106 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:47:35.559073 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:47:41.389233 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:41.389178 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" event={"ID":"bc7451df-e714-4f63-a696-c3078b0f7818","Type":"ContainerStarted","Data":"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14"} Nov 03 08:47:41.389628 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:41.389268 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:42.383380 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:42.383351 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:47.399395 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:47.399310 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:47.520392 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:47.520338 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" podStartSLOduration=8.285933359 podStartE2EDuration="14.520324769s" totalImagesPullingTime="6.23439141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:41.507238669 +0000 UTC m=+485.175279407" watchObservedRunningTime="2026-11-03 08:47:47.520324769 +0000 UTC m=+491.188365507" Nov 03 08:47:47.893605 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:47.893573 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:48.146237 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.146165 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:48.362628 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.362598 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb"] Nov 03 08:47:48.365567 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.365549 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.471207 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.471126 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb"] Nov 03 08:47:48.471554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.471413 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-metrics-certs\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.471554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.471444 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/df03d5cf-0aa6-4f48-aa72-eac45b12c185-kube-api-access-p7s4c\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.471554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.471461 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df03d5cf-0aa6-4f48-aa72-eac45b12c185-openshift-service-ca\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.471658 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.471584 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-maas-api-tls\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.572397 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.572372 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-maas-api-tls\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.572585 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.572432 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-metrics-certs\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.572585 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.572466 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/df03d5cf-0aa6-4f48-aa72-eac45b12c185-kube-api-access-p7s4c\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.572585 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.572484 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df03d5cf-0aa6-4f48-aa72-eac45b12c185-openshift-service-ca\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.573192 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.573126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df03d5cf-0aa6-4f48-aa72-eac45b12c185-openshift-service-ca\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.574902 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.574880 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-maas-api-tls\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.575004 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.574917 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df03d5cf-0aa6-4f48-aa72-eac45b12c185-metrics-certs\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.631751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.631730 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s4c\" (UniqueName: \"kubernetes.io/projected/df03d5cf-0aa6-4f48-aa72-eac45b12c185-kube-api-access-p7s4c\") pod \"maas-api-75bf4769fd-bcvvb\" (UID: \"df03d5cf-0aa6-4f48-aa72-eac45b12c185\") " pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:48.677727 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:48.677702 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:49.051616 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:49.051589 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb"] Nov 03 08:47:49.052533 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:47:49.052508 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf03d5cf_0aa6_4f48_aa72_eac45b12c185.slice/crio-5c504e17e4f70548ce89955d842d2d1645c0909871f180e0729bd7910510f3f6 WatchSource:0}: Error finding container 5c504e17e4f70548ce89955d842d2d1645c0909871f180e0729bd7910510f3f6: Status 404 returned error can't find the container with id 5c504e17e4f70548ce89955d842d2d1645c0909871f180e0729bd7910510f3f6 Nov 03 08:47:49.424261 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:49.424227 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" event={"ID":"df03d5cf-0aa6-4f48-aa72-eac45b12c185","Type":"ContainerStarted","Data":"5c504e17e4f70548ce89955d842d2d1645c0909871f180e0729bd7910510f3f6"} Nov 03 08:47:50.430331 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:50.430304 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" event={"ID":"df03d5cf-0aa6-4f48-aa72-eac45b12c185","Type":"ContainerStarted","Data":"21d9ddd9c121ef090e5cfecf28dd39affee393b6b0f9d7f2025519c0f0db83c2"} Nov 03 08:47:50.430690 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:50.430371 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:50.548527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:50.548484 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" podStartSLOduration=1.6742696719999999 podStartE2EDuration="2.548470241s" totalImagesPullingTime="874.200569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:50.547718569 +0000 UTC m=+494.215759307" watchObservedRunningTime="2026-11-03 08:47:50.548470241 +0000 UTC m=+494.216510978" Nov 03 08:47:52.480582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:52.480528 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:47:52.481075 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:52.480791 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" podUID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" containerName="authorino" containerID="cri-o://1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88" gracePeriod=30 Nov 03 08:47:52.723761 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:52.723739 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:47:52.910424 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:52.910394 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\" (UniqueName: \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\") pod \"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618\" (UID: \"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618\") " Nov 03 08:47:52.912818 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:52.912789 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d" pod "e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" (UID: "e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618"). InnerVolumeSpecName "kube-api-access-5xh9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:47:53.011987 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.011956 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xh9d\" (UniqueName: \"kubernetes.io/projected/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618-kube-api-access-5xh9d\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:53.249350 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.249272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:53.312732 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.312700 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n64sg" Nov 03 08:47:53.377379 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.377346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:47:53.446861 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.446823 4543 generic.go:356] "Generic (PLEG): container finished" podID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" containerID="1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88" exitCode=0 Nov 03 08:47:53.447034 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.446905 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" event={"ID":"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618","Type":"ContainerDied","Data":"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88"} Nov 03 08:47:53.447034 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.446932 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" event={"ID":"e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618","Type":"ContainerDied","Data":"5dc99ce8530b1f7918fdb3be545fdd2378c4d61c091e11f22a517675ca5d23ef"} Nov 03 08:47:53.447034 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.446947 4543 scope.go:122] "RemoveContainer" containerID="1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88" Nov 03 08:47:53.447221 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.447048 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-2b8cr" Nov 03 08:47:53.459073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.459056 4543 scope.go:122] "RemoveContainer" containerID="1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88" Nov 03 08:47:53.459365 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:47:53.459341 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88\": container with ID starting with 1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88 not found: ID does not exist" containerID="1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88" Nov 03 08:47:53.459462 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.459370 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88"} err="failed to get container status \"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88\": rpc error: code = NotFound desc = could not find container \"1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88\": container with ID starting with 1cfc226fff7fb3f2525af7b06b59ea7317d5934a008b9b7b74ad6958b3c4fe88 not found: ID does not exist" Nov 03 08:47:53.617068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.617038 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:47:53.622520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:53.622497 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-2b8cr"] Nov 03 08:47:54.450115 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:54.450084 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" path="/var/lib/kubelet/pods/e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618/volumes" Nov 03 08:47:56.441170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.441129 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75bf4769fd-bcvvb" Nov 03 08:47:56.610505 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.610471 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:56.610738 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.610713 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" podUID="bc7451df-e714-4f63-a696-c3078b0f7818" containerName="maas-api" containerID="cri-o://146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14" gracePeriod=30 Nov 03 08:47:56.881157 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.881113 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:56.948984 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.948958 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\") pod \"bc7451df-e714-4f63-a696-c3078b0f7818\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " Nov 03 08:47:56.949100 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.949003 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\" (UniqueName: \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\") pod \"bc7451df-e714-4f63-a696-c3078b0f7818\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " Nov 03 08:47:56.949100 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.949050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\") pod \"bc7451df-e714-4f63-a696-c3078b0f7818\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " Nov 03 08:47:56.949203 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.949190 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\") pod \"bc7451df-e714-4f63-a696-c3078b0f7818\" (UID: \"bc7451df-e714-4f63-a696-c3078b0f7818\") " Nov 03 08:47:56.949517 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.949488 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca" pod "bc7451df-e714-4f63-a696-c3078b0f7818" (UID: "bc7451df-e714-4f63-a696-c3078b0f7818"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:47:56.951416 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.951395 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs" pod "bc7451df-e714-4f63-a696-c3078b0f7818" (UID: "bc7451df-e714-4f63-a696-c3078b0f7818"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:47:56.951511 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.951450 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls" pod "bc7451df-e714-4f63-a696-c3078b0f7818" (UID: "bc7451df-e714-4f63-a696-c3078b0f7818"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:47:56.951511 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:56.951458 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds" pod "bc7451df-e714-4f63-a696-c3078b0f7818" (UID: "bc7451df-e714-4f63-a696-c3078b0f7818"). InnerVolumeSpecName "kube-api-access-xmhds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:47:57.049933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.049855 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:57.049933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.049882 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7451df-e714-4f63-a696-c3078b0f7818-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:57.049933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.049895 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhds\" (UniqueName: \"kubernetes.io/projected/bc7451df-e714-4f63-a696-c3078b0f7818-kube-api-access-xmhds\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:57.049933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.049908 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7451df-e714-4f63-a696-c3078b0f7818-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:47:57.466389 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.466357 4543 generic.go:356] "Generic (PLEG): container finished" podID="bc7451df-e714-4f63-a696-c3078b0f7818" containerID="146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14" exitCode=0 Nov 03 08:47:57.466788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.466398 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" Nov 03 08:47:57.466788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.466419 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" event={"ID":"bc7451df-e714-4f63-a696-c3078b0f7818","Type":"ContainerDied","Data":"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14"} Nov 03 08:47:57.466788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.466455 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl" event={"ID":"bc7451df-e714-4f63-a696-c3078b0f7818","Type":"ContainerDied","Data":"a06d97d47a53af806547394becdb88d361530016fbe7f2d9e206e4616f4dc888"} Nov 03 08:47:57.466788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.466475 4543 scope.go:122] "RemoveContainer" containerID="146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14" Nov 03 08:47:57.475440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.475424 4543 scope.go:122] "RemoveContainer" containerID="146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14" Nov 03 08:47:57.475690 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:47:57.475675 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14\": container with ID starting with 146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14 not found: ID does not exist" containerID="146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14" Nov 03 08:47:57.475736 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.475696 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14"} err="failed to get container status \"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14\": rpc error: code = NotFound desc = could not find container \"146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14\": container with ID starting with 146a87ef6509aa6102da1b8c0bdde55546aacbb9937fa87b2d3a81624eefeb14 not found: ID does not exist" Nov 03 08:47:57.634697 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.634671 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:57.639956 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:57.639935 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67bdd9df7-89nkl"] Nov 03 08:47:58.451479 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:47:58.451444 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7451df-e714-4f63-a696-c3078b0f7818" path="/var/lib/kubelet/pods/bc7451df-e714-4f63-a696-c3078b0f7818/volumes" Nov 03 08:48:03.798225 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:03.798175 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:48:03.925540 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:03.925512 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:48:05.559715 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:48:05.559683 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:48:18.433213 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:18.433177 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:48:19.821528 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:19.821472 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:48:35.559638 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:48:35.559592 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:48:36.741561 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:36.741488 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:48:36.880995 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:48:36.880965 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:05.558735 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:49:05.558696 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:49:08.425158 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.425109 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:08.450901 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.450871 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:08.451218 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451206 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bc7451df-e714-4f63-a696-c3078b0f7818" containerName="maas-api" Nov 03 08:49:08.451276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451219 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7451df-e714-4f63-a696-c3078b0f7818" containerName="maas-api" Nov 03 08:49:08.451276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451232 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" containerName="authorino" Nov 03 08:49:08.451276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451236 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" containerName="authorino" Nov 03 08:49:08.451377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451289 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7451df-e714-4f63-a696-c3078b0f7818" containerName="maas-api" Nov 03 08:49:08.451377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.451296 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e8eb18-1ab5-4bb7-8e8a-5393c9dae618" containerName="authorino" Nov 03 08:49:08.454105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.454080 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.558216 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.558183 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 08:49:08.558376 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.558291 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 08:49:08.558376 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.558316 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rsqzf\"" Nov 03 08:49:08.559830 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.559809 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:08.576549 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.576523 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw9p6\" (UniqueName: \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.576696 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.576572 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.677337 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.677271 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw9p6\" (UniqueName: \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.677337 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.677319 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.680101 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.680078 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.738173 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.738131 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw9p6\" (UniqueName: \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\") pod \"ai-gateway-controller-5d8dddb676-hs2cc\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.769809 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.769788 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:08.782825 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:08.782793 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:09.142076 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:09.142044 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:09.144492 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:09.144460 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f24e02_6742_435b_af90_a9371f061202.slice/crio-d60e305800f83aeb20e0a436060e058d9f7bcd3a7e56e87d92c3789d448c16ae WatchSource:0}: Error finding container d60e305800f83aeb20e0a436060e058d9f7bcd3a7e56e87d92c3789d448c16ae: Status 404 returned error can't find the container with id d60e305800f83aeb20e0a436060e058d9f7bcd3a7e56e87d92c3789d448c16ae Nov 03 08:49:09.761538 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:09.761503 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" event={"ID":"62f24e02-6742-435b-af90-a9371f061202","Type":"ContainerStarted","Data":"d60e305800f83aeb20e0a436060e058d9f7bcd3a7e56e87d92c3789d448c16ae"} Nov 03 08:49:11.772232 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:11.772199 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" event={"ID":"62f24e02-6742-435b-af90-a9371f061202","Type":"ContainerStarted","Data":"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9"} Nov 03 08:49:11.772643 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:11.772321 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:11.889524 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:11.889478 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" podStartSLOduration=2.056098341 podStartE2EDuration="3.889465867s" totalImagesPullingTime="1.833367526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:11.888367612 +0000 UTC m=+575.556408350" watchObservedRunningTime="2026-11-03 08:49:11.889465867 +0000 UTC m=+575.557506604" Nov 03 08:49:14.895441 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:14.895411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:15.080859 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.080830 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:15.323253 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.323184 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 08:49:15.326516 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.326500 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.384334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.384313 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 08:49:15.432501 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.432477 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 08:49:15.432641 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.432525 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c7jcr\"" Nov 03 08:49:15.434803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.434789 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9lg\" (UniqueName: \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.434882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.434824 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.484520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.484500 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 08:49:15.487733 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.487720 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.519686 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.519664 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 08:49:15.535882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.535856 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9lg\" (UniqueName: \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.535992 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.535909 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.536558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.536536 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.595327 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.595276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9lg\" (UniqueName: \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\") pod \"payload-pre-processing-7b98bb678c-r2s2f\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.636781 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.636757 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxn2s\" (UniqueName: \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.636864 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.636797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.638728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.638711 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:15.737737 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.737707 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxn2s\" (UniqueName: \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.737885 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.737754 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.738486 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.738466 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.796948 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.796921 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxn2s\" (UniqueName: \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\") pod \"payload-processing-55fbbbf9dc-lj9bz\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:15.799760 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:15.799743 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:16.012038 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:16.012014 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 08:49:16.013957 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:16.013928 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4f8360_0c88_4514_83c5_797250d22890.slice/crio-d352c1578713d7fcae480d0011c2a1fe88dd0a03dd23527da0ddd55f5e08ecf7 WatchSource:0}: Error finding container d352c1578713d7fcae480d0011c2a1fe88dd0a03dd23527da0ddd55f5e08ecf7: Status 404 returned error can't find the container with id d352c1578713d7fcae480d0011c2a1fe88dd0a03dd23527da0ddd55f5e08ecf7 Nov 03 08:49:16.169730 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:16.169702 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 08:49:16.172076 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:16.172047 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6ed322_bd89_4c55_b650_28d37f9f208e.slice/crio-3f1c762662796e2b81953fd58784826ca3b03bf1e32c6cdb961da571d0cd61be WatchSource:0}: Error finding container 3f1c762662796e2b81953fd58784826ca3b03bf1e32c6cdb961da571d0cd61be: Status 404 returned error can't find the container with id 3f1c762662796e2b81953fd58784826ca3b03bf1e32c6cdb961da571d0cd61be Nov 03 08:49:16.799000 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:16.798968 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" event={"ID":"6b6ed322-bd89-4c55-b650-28d37f9f208e","Type":"ContainerStarted","Data":"3f1c762662796e2b81953fd58784826ca3b03bf1e32c6cdb961da571d0cd61be"} Nov 03 08:49:16.800082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:16.800057 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" event={"ID":"8d4f8360-0c88-4514-83c5-797250d22890","Type":"ContainerStarted","Data":"d352c1578713d7fcae480d0011c2a1fe88dd0a03dd23527da0ddd55f5e08ecf7"} Nov 03 08:49:18.770777 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:18.770753 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:18.810226 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:18.810200 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" event={"ID":"8d4f8360-0c88-4514-83c5-797250d22890","Type":"ContainerStarted","Data":"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c"} Nov 03 08:49:18.810414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:18.810400 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:18.927935 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:18.927882 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" podStartSLOduration=1.221045942 podStartE2EDuration="3.927863063s" totalImagesPullingTime="2.706817121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:18.926594678 +0000 UTC m=+582.594635416" watchObservedRunningTime="2026-11-03 08:49:18.927863063 +0000 UTC m=+582.595903808" Nov 03 08:49:19.816288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:19.816252 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" event={"ID":"6b6ed322-bd89-4c55-b650-28d37f9f208e","Type":"ContainerStarted","Data":"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d"} Nov 03 08:49:19.816748 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:19.816470 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:19.933536 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:19.933483 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" podStartSLOduration=2.309149104 podStartE2EDuration="4.933468731s" totalImagesPullingTime="2.624319627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:19.931924516 +0000 UTC m=+583.599965256" watchObservedRunningTime="2026-11-03 08:49:19.933468731 +0000 UTC m=+583.601509511" Nov 03 08:49:20.160704 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:20.160670 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:22.779130 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:22.779100 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:23.111059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.111038 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-58b44ff57b-z4gsf"] Nov 03 08:49:23.114555 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.114540 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.171520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.171494 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58b44ff57b-z4gsf"] Nov 03 08:49:23.201513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.201492 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46eb3d7-9990-4e70-b44b-be4a24105698-metrics-certs\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.201648 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.201530 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvjf\" (UniqueName: \"kubernetes.io/projected/e46eb3d7-9990-4e70-b44b-be4a24105698-kube-api-access-npvjf\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.302351 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.302322 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46eb3d7-9990-4e70-b44b-be4a24105698-metrics-certs\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.302480 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.302366 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvjf\" (UniqueName: \"kubernetes.io/projected/e46eb3d7-9990-4e70-b44b-be4a24105698-kube-api-access-npvjf\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.304874 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.304848 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46eb3d7-9990-4e70-b44b-be4a24105698-metrics-certs\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.362501 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.362447 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvjf\" (UniqueName: \"kubernetes.io/projected/e46eb3d7-9990-4e70-b44b-be4a24105698-kube-api-access-npvjf\") pod \"ai-gateway-controller-58b44ff57b-z4gsf\" (UID: \"e46eb3d7-9990-4e70-b44b-be4a24105698\") " pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.426359 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.426338 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:23.796053 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.796028 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58b44ff57b-z4gsf"] Nov 03 08:49:23.798485 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:23.798454 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46eb3d7_9990_4e70_b44b_be4a24105698.slice/crio-a930a70a3d77dc0d9aa55c5240c677c37e1aec1f965ca63685e9f8e9625b80a5 WatchSource:0}: Error finding container a930a70a3d77dc0d9aa55c5240c677c37e1aec1f965ca63685e9f8e9625b80a5: Status 404 returned error can't find the container with id a930a70a3d77dc0d9aa55c5240c677c37e1aec1f965ca63685e9f8e9625b80a5 Nov 03 08:49:23.836210 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:23.836180 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" event={"ID":"e46eb3d7-9990-4e70-b44b-be4a24105698","Type":"ContainerStarted","Data":"a930a70a3d77dc0d9aa55c5240c677c37e1aec1f965ca63685e9f8e9625b80a5"} Nov 03 08:49:24.842491 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:24.842412 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" event={"ID":"e46eb3d7-9990-4e70-b44b-be4a24105698","Type":"ContainerStarted","Data":"1019fb0778519f31433c86b38b6ad56c8514ad2c7666d0bbc983639172974d5f"} Nov 03 08:49:24.842855 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:24.842536 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:24.961367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:24.961320 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" podStartSLOduration=1.188649658 podStartE2EDuration="1.961305979s" totalImagesPullingTime="772.656321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:24.959427742 +0000 UTC m=+588.627468480" watchObservedRunningTime="2026-11-03 08:49:24.961305979 +0000 UTC m=+588.629346716" Nov 03 08:49:29.819621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:29.819591 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 08:49:30.824359 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:30.824333 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 08:49:35.558639 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:49:35.558603 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:49:35.849317 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:35.849252 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-58b44ff57b-z4gsf" Nov 03 08:49:36.022227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.022197 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:36.022429 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.022383 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" podUID="62f24e02-6742-435b-af90-a9371f061202" containerName="manager" containerID="cri-o://e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9" gracePeriod=30 Nov 03 08:49:36.272875 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.272847 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:36.300793 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.300766 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\") pod \"62f24e02-6742-435b-af90-a9371f061202\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " Nov 03 08:49:36.300927 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.300844 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\" (UniqueName: \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\") pod \"62f24e02-6742-435b-af90-a9371f061202\" (UID: \"62f24e02-6742-435b-af90-a9371f061202\") " Nov 03 08:49:36.302887 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.302859 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6" pod "62f24e02-6742-435b-af90-a9371f061202" (UID: "62f24e02-6742-435b-af90-a9371f061202"). InnerVolumeSpecName "kube-api-access-vw9p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:49:36.302887 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.302867 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs" pod "62f24e02-6742-435b-af90-a9371f061202" (UID: "62f24e02-6742-435b-af90-a9371f061202"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:49:36.402098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.402036 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f24e02-6742-435b-af90-a9371f061202-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:49:36.402098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.402071 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw9p6\" (UniqueName: \"kubernetes.io/projected/62f24e02-6742-435b-af90-a9371f061202-kube-api-access-vw9p6\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:49:36.489864 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.489838 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:49:36.490763 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.490740 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:49:36.896495 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.896463 4543 generic.go:356] "Generic (PLEG): container finished" podID="62f24e02-6742-435b-af90-a9371f061202" containerID="e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9" exitCode=0 Nov 03 08:49:36.896962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.896505 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" event={"ID":"62f24e02-6742-435b-af90-a9371f061202","Type":"ContainerDied","Data":"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9"} Nov 03 08:49:36.896962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.896542 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" event={"ID":"62f24e02-6742-435b-af90-a9371f061202","Type":"ContainerDied","Data":"d60e305800f83aeb20e0a436060e058d9f7bcd3a7e56e87d92c3789d448c16ae"} Nov 03 08:49:36.896962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.896562 4543 scope.go:122] "RemoveContainer" containerID="e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9" Nov 03 08:49:36.896962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.896506 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hs2cc" Nov 03 08:49:36.904857 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.904842 4543 scope.go:122] "RemoveContainer" containerID="e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9" Nov 03 08:49:36.905123 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:49:36.905106 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9\": container with ID starting with e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9 not found: ID does not exist" containerID="e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9" Nov 03 08:49:36.905247 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:36.905133 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9"} err="failed to get container status \"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9\": rpc error: code = NotFound desc = could not find container \"e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9\": container with ID starting with e65c5ccaee999ef236981840ccc39a33d494ad93290752d3647e095f0e5160b9 not found: ID does not exist" Nov 03 08:49:37.065830 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:37.065804 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:37.115722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:37.115696 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hs2cc"] Nov 03 08:49:37.582312 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:37.582280 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:38.046508 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:38.046477 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:38.450008 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:38.449971 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f24e02-6742-435b-af90-a9371f061202" path="/var/lib/kubelet/pods/62f24e02-6742-435b-af90-a9371f061202/volumes" Nov 03 08:49:39.213549 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:39.213518 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:39.361377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:39.361351 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:48.551201 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.551165 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg"] Nov 03 08:49:48.551705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.551686 4543 cpu_manager.go:414] "containerMap: removing container" podUID="62f24e02-6742-435b-af90-a9371f061202" containerName="manager" Nov 03 08:49:48.551782 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.551708 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f24e02-6742-435b-af90-a9371f061202" containerName="manager" Nov 03 08:49:48.551831 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.551823 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f24e02-6742-435b-af90-a9371f061202" containerName="manager" Nov 03 08:49:48.557089 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.557070 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595481 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595452 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595609 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595553 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595609 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595581 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtnns\" (UniqueName: \"kubernetes.io/projected/552cd4fb-443b-418b-a382-f9af70e6635f-kube-api-access-mtnns\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595609 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595604 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595764 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595730 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.595818 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.595789 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552cd4fb-443b-418b-a382-f9af70e6635f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.616710 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.616683 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg"] Nov 03 08:49:48.641960 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.641935 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:48.664620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.664598 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 08:49:48.664728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.664617 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 08:49:48.664728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.664600 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7rrjj\"" Nov 03 08:49:48.664728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.664648 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 08:49:48.696402 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696372 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696450 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696476 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtnns\" (UniqueName: \"kubernetes.io/projected/552cd4fb-443b-418b-a382-f9af70e6635f-kube-api-access-mtnns\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696538 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696597 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552cd4fb-443b-418b-a382-f9af70e6635f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696767 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696753 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.696901 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.696881 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.697171 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.697128 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.698668 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.698645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/552cd4fb-443b-418b-a382-f9af70e6635f-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.698942 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.698926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552cd4fb-443b-418b-a382-f9af70e6635f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.759417 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.759384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtnns\" (UniqueName: \"kubernetes.io/projected/552cd4fb-443b-418b-a382-f9af70e6635f-kube-api-access-mtnns\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg\" (UID: \"552cd4fb-443b-418b-a382-f9af70e6635f\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:48.869665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:48.869637 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:49.240408 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:49.240387 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg"] Nov 03 08:49:49.242917 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:49.242894 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552cd4fb_443b_418b_a382_f9af70e6635f.slice/crio-8b4178f386368a10add80d2f63a708ea8c088d814ca79c27173031bc8c62d431 WatchSource:0}: Error finding container 8b4178f386368a10add80d2f63a708ea8c088d814ca79c27173031bc8c62d431: Status 404 returned error can't find the container with id 8b4178f386368a10add80d2f63a708ea8c088d814ca79c27173031bc8c62d431 Nov 03 08:49:49.244215 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:49.244198 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 08:49:49.960669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:49.960639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" event={"ID":"552cd4fb-443b-418b-a382-f9af70e6635f","Type":"ContainerStarted","Data":"8b4178f386368a10add80d2f63a708ea8c088d814ca79c27173031bc8c62d431"} Nov 03 08:49:51.970347 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:51.970304 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" event={"ID":"552cd4fb-443b-418b-a382-f9af70e6635f","Type":"ContainerStarted","Data":"6127b78e8c4ebc4e5138a6776e7e98bd5e2cc006b6444e08b178943e45888387"} Nov 03 08:49:52.091338 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:52.091292 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" podStartSLOduration=1.9492518049999998 podStartE2EDuration="4.091279087s" totalImagesPullingTime="2.142027282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:52.090399007 +0000 UTC m=+615.758439744" watchObservedRunningTime="2026-11-03 08:49:52.091279087 +0000 UTC m=+615.759319819" Nov 03 08:49:52.869959 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:52.869928 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:55.432980 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.432946 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm"] Nov 03 08:49:55.436502 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.436487 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.540682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.540657 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 08:49:55.542327 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.542300 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:49:55.542774 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.542755 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm"] Nov 03 08:49:55.556601 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556577 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.556726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556620 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b664a803-96f3-43fc-bf22-fb740e6bbef7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.556726 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556696 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzs7j\" (UniqueName: \"kubernetes.io/projected/b664a803-96f3-43fc-bf22-fb740e6bbef7-kube-api-access-qzs7j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.556827 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.556876 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556845 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.556876 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.556860 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.657930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.657904 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b664a803-96f3-43fc-bf22-fb740e6bbef7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.657937 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzs7j\" (UniqueName: \"kubernetes.io/projected/b664a803-96f3-43fc-bf22-fb740e6bbef7-kube-api-access-qzs7j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.657994 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658040 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658064 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658114 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658483 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658461 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658575 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658527 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.658575 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.658543 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.660264 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.660244 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b664a803-96f3-43fc-bf22-fb740e6bbef7-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.660443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.660427 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b664a803-96f3-43fc-bf22-fb740e6bbef7-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.718039 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.717984 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzs7j\" (UniqueName: \"kubernetes.io/projected/b664a803-96f3-43fc-bf22-fb740e6bbef7-kube-api-access-qzs7j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-2nzfm\" (UID: \"b664a803-96f3-43fc-bf22-fb740e6bbef7\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:55.748851 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:55.748830 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:56.117983 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:56.117964 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm"] Nov 03 08:49:56.120287 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:49:56.120261 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb664a803_96f3_43fc_bf22_fb740e6bbef7.slice/crio-e5f77f46d13a9ae04f1a7dfc5aa40539be04934fc4e3d13b648511720ef499e5 WatchSource:0}: Error finding container e5f77f46d13a9ae04f1a7dfc5aa40539be04934fc4e3d13b648511720ef499e5: Status 404 returned error can't find the container with id e5f77f46d13a9ae04f1a7dfc5aa40539be04934fc4e3d13b648511720ef499e5 Nov 03 08:49:56.992932 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:56.992846 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" event={"ID":"b664a803-96f3-43fc-bf22-fb740e6bbef7","Type":"ContainerStarted","Data":"f00c0389b89686dbe4c3a98e55906ea63ee161d3a1adfb6e163f9560d11d9fed"} Nov 03 08:49:56.992932 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:56.992894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" event={"ID":"b664a803-96f3-43fc-bf22-fb740e6bbef7","Type":"ContainerStarted","Data":"e5f77f46d13a9ae04f1a7dfc5aa40539be04934fc4e3d13b648511720ef499e5"} Nov 03 08:49:57.110863 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:57.110818 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" podStartSLOduration=1.510002544 podStartE2EDuration="2.110805867s" totalImagesPullingTime="600.803323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:49:57.110233952 +0000 UTC m=+620.778274690" watchObservedRunningTime="2026-11-03 08:49:57.110805867 +0000 UTC m=+620.778846604" Nov 03 08:49:57.748954 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:57.748925 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:49:58.870085 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:58.870053 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:58.882349 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:58.882329 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:59.012854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:59.012828 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg" Nov 03 08:49:59.399919 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:49:59.399891 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:03.375889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.375858 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj"] Nov 03 08:50:03.379412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.379396 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.439880 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.439853 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj"] Nov 03 08:50:03.470432 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.470399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:03.486728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.486683 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 08:50:03.522072 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522046 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.522224 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522077 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.522224 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522096 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.522224 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522176 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.522347 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522324 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.522412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.522399 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/b22e17fb-2005-42e7-ab1e-31c31d3d7606-kube-api-access-xjfp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623187 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623216 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/b22e17fb-2005-42e7-ab1e-31c31d3d7606-kube-api-access-xjfp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623271 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623290 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623343 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623675 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623747 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623725 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.623810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.623786 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.625548 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.625528 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b22e17fb-2005-42e7-ab1e-31c31d3d7606-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.625930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.625884 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b22e17fb-2005-42e7-ab1e-31c31d3d7606-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.682490 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.682467 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfp2\" (UniqueName: \"kubernetes.io/projected/b22e17fb-2005-42e7-ab1e-31c31d3d7606-kube-api-access-xjfp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xtttj\" (UID: \"b22e17fb-2005-42e7-ab1e-31c31d3d7606\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:03.691312 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:03.691288 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:04.058868 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:04.058847 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj"] Nov 03 08:50:04.060939 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:50:04.060912 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22e17fb_2005_42e7_ab1e_31c31d3d7606.slice/crio-46647dc79ae894059434a24ae98b10622b93ce534e48c1f60c3a84e118f11d29 WatchSource:0}: Error finding container 46647dc79ae894059434a24ae98b10622b93ce534e48c1f60c3a84e118f11d29: Status 404 returned error can't find the container with id 46647dc79ae894059434a24ae98b10622b93ce534e48c1f60c3a84e118f11d29 Nov 03 08:50:05.030436 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.030349 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" event={"ID":"b22e17fb-2005-42e7-ab1e-31c31d3d7606","Type":"ContainerStarted","Data":"5d3cb6f8075ca472d962ed213a207ad7ef0269b3a65f7d2041766b5a23d2464c"} Nov 03 08:50:05.030436 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.030406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" event={"ID":"b22e17fb-2005-42e7-ab1e-31c31d3d7606","Type":"ContainerStarted","Data":"46647dc79ae894059434a24ae98b10622b93ce534e48c1f60c3a84e118f11d29"} Nov 03 08:50:05.149033 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.148980 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" podStartSLOduration=1.5524389219999999 podStartE2EDuration="2.148965764s" totalImagesPullingTime="596.526842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:05.148201794 +0000 UTC m=+628.816242533" watchObservedRunningTime="2026-11-03 08:50:05.148965764 +0000 UTC m=+628.817006502" Nov 03 08:50:05.559672 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:50:05.559631 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:50:05.692407 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.692380 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:05.749223 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.749184 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:50:05.761449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:05.761426 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:50:06.045600 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:06.045569 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-2nzfm" Nov 03 08:50:08.210347 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:08.210314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:12.461810 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.461784 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tq9b2"] Nov 03 08:50:12.465282 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.465266 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.520288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.520261 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 08:50:12.559775 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.559749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:12.571566 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.571543 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tq9b2"] Nov 03 08:50:12.597964 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.597943 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnrmv\" (UniqueName: \"kubernetes.io/projected/d06e15c0-3989-40f0-b3b8-dfb108624b4e-kube-api-access-cnrmv\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.598060 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.597988 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.598227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.598212 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.598292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.598280 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.598339 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.598326 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.598390 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.598372 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699308 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699283 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699311 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699337 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnrmv\" (UniqueName: \"kubernetes.io/projected/d06e15c0-3989-40f0-b3b8-dfb108624b4e-kube-api-access-cnrmv\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699374 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699404 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699554 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699776 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699747 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.699776 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699773 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.700097 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.699794 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-home\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.701573 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.701551 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06e15c0-3989-40f0-b3b8-dfb108624b4e-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.701901 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.701885 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06e15c0-3989-40f0-b3b8-dfb108624b4e-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.758340 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.758291 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnrmv\" (UniqueName: \"kubernetes.io/projected/d06e15c0-3989-40f0-b3b8-dfb108624b4e-kube-api-access-cnrmv\") pod \"echo-simulator-kserve-64b6f8f96c-tq9b2\" (UID: \"d06e15c0-3989-40f0-b3b8-dfb108624b4e\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:12.777328 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:12.777309 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:13.147573 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:13.147553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tq9b2"] Nov 03 08:50:13.148823 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:50:13.148799 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06e15c0_3989_40f0_b3b8_dfb108624b4e.slice/crio-89fc872be53603a0efc9dfcf02560a351ba39ae35b997d8db312d91825c86949 WatchSource:0}: Error finding container 89fc872be53603a0efc9dfcf02560a351ba39ae35b997d8db312d91825c86949: Status 404 returned error can't find the container with id 89fc872be53603a0efc9dfcf02560a351ba39ae35b997d8db312d91825c86949 Nov 03 08:50:13.692132 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:13.692106 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:13.704295 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:13.704274 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:14.069880 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:14.069800 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" event={"ID":"d06e15c0-3989-40f0-b3b8-dfb108624b4e","Type":"ContainerStarted","Data":"89fc872be53603a0efc9dfcf02560a351ba39ae35b997d8db312d91825c86949"} Nov 03 08:50:14.080950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:14.080928 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xtttj" Nov 03 08:50:16.088437 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:16.088342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" event={"ID":"d06e15c0-3989-40f0-b3b8-dfb108624b4e","Type":"ContainerStarted","Data":"39431b328ba7098620100cced7022fcc5959fea9a6bd19b05037018092d5a7f8"} Nov 03 08:50:16.209322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:16.209257 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" podStartSLOduration=1.623958121 podStartE2EDuration="4.209240265s" totalImagesPullingTime="2.585282144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:16.208885345 +0000 UTC m=+639.876926084" watchObservedRunningTime="2026-11-03 08:50:16.209240265 +0000 UTC m=+639.877281004" Nov 03 08:50:16.777548 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:16.777510 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:17.424485 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:17.424452 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:19.114634 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:19.114599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:20.521199 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:20.521164 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:21.443677 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:21.443644 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:22.778394 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:22.778366 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:22.790430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:22.790407 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:23.133315 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:23.133286 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tq9b2" Nov 03 08:50:26.196848 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:26.196819 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:26.327133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:26.327094 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:32.595529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.595500 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 08:50:32.607321 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.607295 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.662412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.662387 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 08:50:32.670378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670352 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670507 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670383 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670507 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdfjr\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670578 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670551 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670612 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670581 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670668 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670653 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670703 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670675 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670743 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670716 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.670835 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.670814 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.712151 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.712106 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\"" Nov 03 08:50:32.714186 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.714160 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 08:50:32.771421 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771397 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771421 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771423 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771602 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771458 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771602 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771475 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771602 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771501 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771754 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771632 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771754 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771741 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdfjr\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771849 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771775 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771849 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771814 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771948 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771845 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.771948 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771879 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.772055 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.771981 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.772193 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.772171 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.772408 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.772392 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.773882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.773854 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.774104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.774081 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.831685 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.831663 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.831784 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.831765 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdfjr\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-65rgg\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:32.920584 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:32.920551 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:33.348228 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:33.348155 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 08:50:33.385918 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:50:33.385893 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0b110a_bc54_4a4b_a48e_11e2383a616d.slice/crio-015092459a167799d7a6f6d5389d7efab0f3ac85bc3b50602ae1a3d9efbd3099 WatchSource:0}: Error finding container 015092459a167799d7a6f6d5389d7efab0f3ac85bc3b50602ae1a3d9efbd3099: Status 404 returned error can't find the container with id 015092459a167799d7a6f6d5389d7efab0f3ac85bc3b50602ae1a3d9efbd3099 Nov 03 08:50:34.173376 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:34.173343 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" event={"ID":"cd0b110a-bc54-4a4b-a48e-11e2383a616d","Type":"ContainerStarted","Data":"015092459a167799d7a6f6d5389d7efab0f3ac85bc3b50602ae1a3d9efbd3099"} Nov 03 08:50:35.568298 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:50:35.568262 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:50:37.130758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:37.130726 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:37.131064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:37.130795 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:37.131064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:37.130827 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:38.209581 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:38.209537 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" event={"ID":"cd0b110a-bc54-4a4b-a48e-11e2383a616d","Type":"ContainerStarted","Data":"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d"} Nov 03 08:50:38.401438 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:38.401374 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podStartSLOduration=2.659229545 podStartE2EDuration="6.40135568s" totalImagesPullingTime="3.742126135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:38.400601728 +0000 UTC m=+662.068642467" watchObservedRunningTime="2026-11-03 08:50:38.40135568 +0000 UTC m=+662.069396418" Nov 03 08:50:38.921116 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:38.921088 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:38.922600 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:38.922568 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:38.922698 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:38.922634 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:39.921645 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:39.921611 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:39.922031 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:39.921664 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:40.921608 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:40.921575 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:40.921807 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:40.921632 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:41.921415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:41.921380 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:41.921582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:41.921428 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:42.921598 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:42.921556 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:42.922007 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:42.921620 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:43.921402 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:43.921368 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 08:50:43.921603 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:43.921425 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 08:50:45.145558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:45.145517 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:45.145992 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:45.145881 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:45.146826 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:45.146804 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 08:50:47.595998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.595921 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:50:47.599631 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.599615 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.648298 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.648271 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 08:50:47.653074 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.653054 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710492 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710469 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710516 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9r47\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710593 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.710742 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710657 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710742 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710689 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.710742 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710723 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.710839 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710790 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710839 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710814 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.710839 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.710961 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710912 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.710961 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.710952 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.711064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711002 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2thb\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.711064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711052 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.711181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711084 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.711181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711121 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.711257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711179 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.711257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.711257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.711252 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.712334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.712317 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:50:47.804293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.804268 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 08:50:47.812406 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812380 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812429 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812461 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9r47\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812482 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812497 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812625 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812650 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812687 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812748 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812783 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812813 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2thb\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812878 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812888 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812908 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.812924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812912 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812946 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.812976 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813011 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813206 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813214 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813222 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813521 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813706 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.813762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813742 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.813762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813743 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.813931 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.813810 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.815769 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.815744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.816103 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.816083 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.816683 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.816657 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.816782 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.816711 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.873840 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.873812 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.874189 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.874172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2thb\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.874529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.874507 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h65zm\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.874944 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.874926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9r47\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-lk9tp\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:47.911430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.911412 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:47.966896 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:47.966405 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:48.345028 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.345001 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:50:48.345978 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:50:48.345950 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d103670_9db2_4e56_8b59_c58812a52487.slice/crio-5c9badd8f2609075f96df881b3d0b366731e6e3f74ab2fab768db2cd0eb27159 WatchSource:0}: Error finding container 5c9badd8f2609075f96df881b3d0b366731e6e3f74ab2fab768db2cd0eb27159: Status 404 returned error can't find the container with id 5c9badd8f2609075f96df881b3d0b366731e6e3f74ab2fab768db2cd0eb27159 Nov 03 08:50:48.348816 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.348786 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.348898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.348847 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.348898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.348875 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.399858 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.399839 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 08:50:48.401636 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:50:48.401610 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f54a4f_fc67_4f41_a7a7_64ffbefea13b.slice/crio-54b9a994731dec30bb0e5048283e9a77a3bee40cb57ee5a9094a2a3cc49d9821 WatchSource:0}: Error finding container 54b9a994731dec30bb0e5048283e9a77a3bee40cb57ee5a9094a2a3cc49d9821: Status 404 returned error can't find the container with id 54b9a994731dec30bb0e5048283e9a77a3bee40cb57ee5a9094a2a3cc49d9821 Nov 03 08:50:48.404423 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.404397 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.404498 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.404453 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.404498 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.404480 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:50:48.581498 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.581469 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:48.724067 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:48.724027 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:49.260473 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.260424 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" event={"ID":"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b","Type":"ContainerStarted","Data":"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236"} Nov 03 08:50:49.260673 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.260516 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" event={"ID":"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b","Type":"ContainerStarted","Data":"54b9a994731dec30bb0e5048283e9a77a3bee40cb57ee5a9094a2a3cc49d9821"} Nov 03 08:50:49.262296 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.262272 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" event={"ID":"4d103670-9db2-4e56-8b59-c58812a52487","Type":"ContainerStarted","Data":"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857"} Nov 03 08:50:49.262425 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.262325 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" event={"ID":"4d103670-9db2-4e56-8b59-c58812a52487","Type":"ContainerStarted","Data":"5c9badd8f2609075f96df881b3d0b366731e6e3f74ab2fab768db2cd0eb27159"} Nov 03 08:50:49.399165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.394409 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podStartSLOduration=2.394389378 podStartE2EDuration="2.394389378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:49.393625644 +0000 UTC m=+673.061666398" watchObservedRunningTime="2026-11-03 08:50:49.394389378 +0000 UTC m=+673.062430117" Nov 03 08:50:49.520224 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.517424 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podStartSLOduration=2.517407228 podStartE2EDuration="2.517407228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:49.516860944 +0000 UTC m=+673.184901683" watchObservedRunningTime="2026-11-03 08:50:49.517407228 +0000 UTC m=+673.185447967" Nov 03 08:50:49.912517 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.912484 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:49.918169 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.917839 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:49.918169 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.917890 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:49.967468 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.967438 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:49.968986 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.968955 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:49.969114 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:49.969002 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:50.015073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.015043 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:50.163779 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.163701 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:50:50.918236 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.918203 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:50.918800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.918770 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:50.967866 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.967833 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:50.968027 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:50.967882 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:51.914163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:51.913476 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:51.914163 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:51.913564 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:51.967688 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:51.967645 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:51.968070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:51.967712 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:52.912591 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:52.912543 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:52.912890 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:52.912863 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:52.967760 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:52.967728 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:52.968230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:52.968202 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:53.915550 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:53.915512 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:53.915828 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:53.915802 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:53.967455 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:53.967413 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:53.967688 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:53.967665 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:54.912020 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:54.911981 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h65zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 08:50:54.912557 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:54.912032 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 08:50:54.967294 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:54.967260 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 03 08:50:54.967438 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:54.967303 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 03 08:50:56.557509 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.557471 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:56.557958 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.557935 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:56.558735 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.558712 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:56.559225 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.559201 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:50:56.559475 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.559454 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:50:56.560242 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:50:56.560224 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 08:51:05.560295 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:51:05.560258 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:51:18.589556 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:18.589518 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" podUID="20fdfbfab5aa17ccf7324ae585a08734" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 08:51:19.463073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:19.463039 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:20.862695 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:20.862661 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:25.523825 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:25.523792 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:30.353308 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:30.353266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:35.559295 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:51:35.559262 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:51:51.109831 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.109796 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:51.288555 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.288523 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:51.444722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.444693 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:51.802088 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.801993 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:51:51.805464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.805446 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.861836 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.861800 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-82eec5e9-openshift-default-dockercfg-wwggq\"" Nov 03 08:51:51.879408 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.879376 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:51.895812 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.895781 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.895951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.895819 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.895951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.895840 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.895951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.895935 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9zm\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.896104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.896003 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.896104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.896034 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.896104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.896062 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.896104 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.896083 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.896325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.896212 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.913090 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.913066 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:51:51.997481 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997446 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997631 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997534 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997631 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997583 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997631 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997790 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997646 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9zm\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997790 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997690 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997790 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997735 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997790 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997782 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997986 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997818 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.997986 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.997843 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.998429 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.998402 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.998567 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.998481 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.998814 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.998795 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:51.998882 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:51.998816 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.000263 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.000240 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.000355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.000339 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.027364 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.027343 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:52.059663 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.059608 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.059663 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.059652 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9zm\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\") pod \"e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.118131 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.118107 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:52.373199 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.373171 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:52.539045 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.539017 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:52.567157 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.567128 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:51:52.569295 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:51:52.569271 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57713756_4173_4d37_b2e3_7d914e603247.slice/crio-b7a249cdb149ee609dc084136ed1b22172f3a046ecbde23eaf930b4539916b37 WatchSource:0}: Error finding container b7a249cdb149ee609dc084136ed1b22172f3a046ecbde23eaf930b4539916b37: Status 404 returned error can't find the container with id b7a249cdb149ee609dc084136ed1b22172f3a046ecbde23eaf930b4539916b37 Nov 03 08:51:52.571395 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.571366 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:52.571484 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.571426 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:52.571484 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.571453 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:52.898615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:52.898539 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:53.564197 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.564163 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerStarted","Data":"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25"} Nov 03 08:51:53.564666 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.564224 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerStarted","Data":"b7a249cdb149ee609dc084136ed1b22172f3a046ecbde23eaf930b4539916b37"} Nov 03 08:51:53.685907 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.685855 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podStartSLOduration=2.685842351 podStartE2EDuration="2.685842351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:51:53.684068336 +0000 UTC m=+737.352109103" watchObservedRunningTime="2026-11-03 08:51:53.685842351 +0000 UTC m=+737.353883166" Nov 03 08:51:53.736809 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.736779 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:51:53.740336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.740320 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.795085 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.795059 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6a1442a9-openshift-default-dockercfg-dcrtc\"" Nov 03 08:51:53.800763 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.800741 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:51:53.809953 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.809932 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:53.817196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817134 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817224 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4pp\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817287 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817319 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817351 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817376 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817431 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817472 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.817554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.817511 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.917996 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.917967 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918001 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4pp\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918021 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918131 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918245 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918298 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918353 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918443 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918428 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918642 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918642 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918763 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918705 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.918763 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918763 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.919020 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.918991 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.920506 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.920471 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.920778 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.920763 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.977701 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.977592 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:53.978238 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:53.978218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4pp\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\") pod \"e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:54.041326 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.041255 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:54.058068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.058042 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:54.120892 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.120865 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:54.126240 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.126213 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:54.484948 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.484920 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:51:54.486637 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:51:54.486610 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2742b2f3_a7c0_4428_8d0a_589e67781fe5.slice/crio-0a718003b4dd7adab6d42cb59ed538862a1a2e8a6065c6fba85f690df50e76da WatchSource:0}: Error finding container 0a718003b4dd7adab6d42cb59ed538862a1a2e8a6065c6fba85f690df50e76da: Status 404 returned error can't find the container with id 0a718003b4dd7adab6d42cb59ed538862a1a2e8a6065c6fba85f690df50e76da Nov 03 08:51:54.488777 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.488744 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:54.488842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.488808 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:54.488842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.488831 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:51:54.572020 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.571982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerStarted","Data":"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7"} Nov 03 08:51:54.572450 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.572044 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerStarted","Data":"0a718003b4dd7adab6d42cb59ed538862a1a2e8a6065c6fba85f690df50e76da"} Nov 03 08:51:54.572544 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.572525 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:54.573608 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.573585 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:51:54.693855 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:54.693807 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podStartSLOduration=1.693792185 podStartE2EDuration="1.693792185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:51:54.691025649 +0000 UTC m=+738.359066388" watchObservedRunningTime="2026-11-03 08:51:54.693792185 +0000 UTC m=+738.361832985" Nov 03 08:51:55.058590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:55.058559 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:55.063622 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:55.063599 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:55.577784 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:55.577751 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:55.578500 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:55.578484 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:51:55.988225 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:55.988190 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:56.236883 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:56.236830 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:56.622124 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:56.622098 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:56.827516 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:56.827483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:57.022167 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:57.022045 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:57.281189 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:57.281050 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:57.518979 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:57.518913 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:57.824674 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:57.824628 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:59.429544 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:59.429514 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:59.563081 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:59.563045 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:51:59.738791 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:51:59.738678 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:00.382393 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:00.382334 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:00.557260 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:00.557218 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:01.498004 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:01.497971 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:01.990496 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:01.990466 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 08:52:01.994149 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:01.994125 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.051526 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.051492 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-98d5dd-openshift-default-dockercfg-dt94n\"" Nov 03 08:52:02.088269 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.088237 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:02.099575 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099550 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6z7\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099741 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099594 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099741 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099620 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099741 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099728 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099892 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099820 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099892 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.099998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099969 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.100057 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.099994 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.100101 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.100072 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.104042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.104023 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 08:52:02.107852 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.107835 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:52:02.111407 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.111394 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.168162 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.168118 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-82eec5e9-serving-cert\"" Nov 03 08:52:02.170171 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.170131 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:52:02.200699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200671 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.200828 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200717 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.200868 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200841 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.200922 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200907 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.200979 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200944 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.200979 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200962 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201083 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.200986 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57px\" (UniqueName: \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.201134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201117 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201216 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201130 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.201216 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201195 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201320 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201213 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201320 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201241 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201320 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201474 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6z7\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201474 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201398 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201586 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201473 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201748 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201664 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.201748 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.201714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.203596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.203573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.203683 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.203602 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.262802 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.262723 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.262989 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.262865 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6z7\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.270022 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.269997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:02.302207 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.302178 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.302337 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.302261 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.302337 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.302320 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.302428 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:02.302414 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-serving-cert: secret "maas-api-e2e-ait-82eec5e9-serving-cert" not found Nov 03 08:52:02.302464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.302432 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57px\" (UniqueName: \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.302508 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:02.302482 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls podName:f541447b-2668-42b5-b3ed-1d1faa1ee02f nodeName:}" failed. No retries permitted until 2026-11-03 08:52:02.802462569 +0000 UTC m=+746.470503357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls") pod "maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" (UID: "f541447b-2668-42b5-b3ed-1d1faa1ee02f") : secret "maas-api-e2e-ait-82eec5e9-serving-cert" not found Nov 03 08:52:02.302847 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.302825 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.304651 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.304634 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.306638 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.306618 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:02.365982 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.364808 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57px\" (UniqueName: \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.421949 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.421916 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:02.560679 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.560599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:02.734014 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.733991 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 08:52:02.735848 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:52:02.735811 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fc4824_232a_4853_8f9f_9e183c2159c5.slice/crio-5893c9796373b5baa412607bf4013b00c813c626d2dd420bd3594a0e7f55578d WatchSource:0}: Error finding container 5893c9796373b5baa412607bf4013b00c813c626d2dd420bd3594a0e7f55578d: Status 404 returned error can't find the container with id 5893c9796373b5baa412607bf4013b00c813c626d2dd420bd3594a0e7f55578d Nov 03 08:52:02.737917 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.737887 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:02.737991 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.737946 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:02.737991 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.737977 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:02.808403 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.808377 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:02.810932 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:02.810874 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") pod \"maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:03.025119 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.025089 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:03.404590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.404553 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:52:03.405913 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:52:03.405888 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf541447b_2668_42b5_b3ed_1d1faa1ee02f.slice/crio-2a23db3fcbdfc44e2f27045aeb7240c6d5bfcf3a2333fa5c92f77a814a81131c WatchSource:0}: Error finding container 2a23db3fcbdfc44e2f27045aeb7240c6d5bfcf3a2333fa5c92f77a814a81131c: Status 404 returned error can't find the container with id 2a23db3fcbdfc44e2f27045aeb7240c6d5bfcf3a2333fa5c92f77a814a81131c Nov 03 08:52:03.619607 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.619571 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerStarted","Data":"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611"} Nov 03 08:52:03.619803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.619634 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerStarted","Data":"5893c9796373b5baa412607bf4013b00c813c626d2dd420bd3594a0e7f55578d"} Nov 03 08:52:03.620934 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.620911 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" event={"ID":"f541447b-2668-42b5-b3ed-1d1faa1ee02f","Type":"ContainerStarted","Data":"2a23db3fcbdfc44e2f27045aeb7240c6d5bfcf3a2333fa5c92f77a814a81131c"} Nov 03 08:52:03.740785 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:03.740699 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podStartSLOduration=2.740685928 podStartE2EDuration="2.740685928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:03.739566325 +0000 UTC m=+747.407607064" watchObservedRunningTime="2026-11-03 08:52:03.740685928 +0000 UTC m=+747.408726665" Nov 03 08:52:04.307371 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.307349 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:04.312778 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.312755 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:04.566978 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.566902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:04.627391 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.627359 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" event={"ID":"f541447b-2668-42b5-b3ed-1d1faa1ee02f","Type":"ContainerStarted","Data":"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3"} Nov 03 08:52:04.627605 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.627588 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:04.627667 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.627626 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:04.628513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.628495 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:04.736286 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.736253 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:04.747489 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.747446 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" podStartSLOduration=1.853636659 podStartE2EDuration="2.747432881s" totalImagesPullingTime="893.796222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:04.746704285 +0000 UTC m=+748.414745023" watchObservedRunningTime="2026-11-03 08:52:04.747432881 +0000 UTC m=+748.415473631" Nov 03 08:52:04.877990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:04.877958 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:05.025564 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:05.025532 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:05.560130 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:05.560098 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:52:07.128338 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.128307 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:07.313151 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.313116 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:07.792116 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.792069 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:52:07.796091 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.796071 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.863729 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.863702 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.863873 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.863828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.863912 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.863876 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.863955 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.863928 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9kdf\" (UniqueName: \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.900967 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.900944 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6a1442a9-serving-cert\"" Nov 03 08:52:07.903422 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.903401 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:52:07.964848 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.964824 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.964967 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.964858 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.964967 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.964892 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9kdf\" (UniqueName: \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.964967 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.964923 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.965063 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:07.964977 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-serving-cert: secret "maas-api-e2e-ait-6a1442a9-serving-cert" not found Nov 03 08:52:07.965063 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:07.965049 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls podName:e62b8353-aaef-4a9d-a6d3-ffc6b200664a nodeName:}" failed. No retries permitted until 2026-11-03 08:52:08.465029826 +0000 UTC m=+752.133070549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls") pod "maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" (UID: "e62b8353-aaef-4a9d-a6d3-ffc6b200664a") : secret "maas-api-e2e-ait-6a1442a9-serving-cert" not found Nov 03 08:52:07.965497 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.965478 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:07.967190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:07.967172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:08.026816 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.026787 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9kdf\" (UniqueName: \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:08.046300 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.046228 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:08.196743 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.196695 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:08.368274 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.368241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:08.470764 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.470737 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:08.472962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.472933 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") pod \"maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:08.708828 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.708758 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:08.778410 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.773876 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:08.995792 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:08.995722 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:09.086133 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:09.086109 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:52:09.087953 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:52:09.087916 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62b8353_aaef_4a9d_a6d3_ffc6b200664a.slice/crio-f8f756f9993d1b54db6da3179fa5aac5b95b541d39d6bd8b4583802cdef8b7de WatchSource:0}: Error finding container f8f756f9993d1b54db6da3179fa5aac5b95b541d39d6bd8b4583802cdef8b7de: Status 404 returned error can't find the container with id f8f756f9993d1b54db6da3179fa5aac5b95b541d39d6bd8b4583802cdef8b7de Nov 03 08:52:09.143214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:09.143175 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:09.655069 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:09.655040 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" event={"ID":"e62b8353-aaef-4a9d-a6d3-ffc6b200664a","Type":"ContainerStarted","Data":"f8f756f9993d1b54db6da3179fa5aac5b95b541d39d6bd8b4583802cdef8b7de"} Nov 03 08:52:10.637887 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:10.637853 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:52:10.661209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:10.661172 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" event={"ID":"e62b8353-aaef-4a9d-a6d3-ffc6b200664a","Type":"ContainerStarted","Data":"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c"} Nov 03 08:52:10.661652 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:10.661254 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:10.875504 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:10.875452 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" podStartSLOduration=3.056218468 podStartE2EDuration="3.875438983s" totalImagesPullingTime="819.220515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:10.873606146 +0000 UTC m=+754.541646883" watchObservedRunningTime="2026-11-03 08:52:10.875438983 +0000 UTC m=+754.543479721" Nov 03 08:52:12.125529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:12.125492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:12.295126 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:12.295082 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:13.282288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.282255 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:52:13.285927 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.285911 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.341125 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.341096 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b6ef5ff5-serving-cert\"" Nov 03 08:52:13.392111 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.392082 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:52:13.423891 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.423862 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.424031 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.423919 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmjw\" (UniqueName: \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.424071 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.424037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.424102 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.424082 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.517196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.517165 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:13.525398 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.525373 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmjw\" (UniqueName: \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.525560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.525444 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.525560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.525467 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.525560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.525531 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.525713 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:13.525656 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-serving-cert: secret "maas-api-e2e-ait-b6ef5ff5-serving-cert" not found Nov 03 08:52:13.525759 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:13.525721 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls podName:950f126c-5642-4822-b947-35b7b1838c4d nodeName:}" failed. No retries permitted until 2026-11-03 08:52:14.025702523 +0000 UTC m=+757.693743240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls") pod "maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" (UID: "950f126c-5642-4822-b947-35b7b1838c4d") : secret "maas-api-e2e-ait-b6ef5ff5-serving-cert" not found Nov 03 08:52:13.526364 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.526343 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.527847 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.527822 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.587067 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.586991 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmjw\" (UniqueName: \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:13.679763 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.679730 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:13.807704 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:13.807667 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:14.030314 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:14.030285 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:14.032950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:14.032926 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") pod \"maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:14.198425 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:14.198396 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:14.577705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:14.577680 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:52:14.578868 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:52:14.578842 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950f126c_5642_4822_b947_35b7b1838c4d.slice/crio-0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644 WatchSource:0}: Error finding container 0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644: Status 404 returned error can't find the container with id 0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644 Nov 03 08:52:14.682742 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:14.682711 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" event={"ID":"950f126c-5642-4822-b947-35b7b1838c4d","Type":"ContainerStarted","Data":"0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644"} Nov 03 08:52:15.693490 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:15.693390 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" event={"ID":"950f126c-5642-4822-b947-35b7b1838c4d","Type":"ContainerStarted","Data":"23aeddb0cf409d9a644ba8248a1e766c9c50d35abf5d63bee6f1c1361f3d17c9"} Nov 03 08:52:15.694282 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:15.694252 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:15.813669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:15.813619 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" podStartSLOduration=1.990909196 podStartE2EDuration="2.81360314s" totalImagesPullingTime="822.693944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:15.811615467 +0000 UTC m=+759.479656206" watchObservedRunningTime="2026-11-03 08:52:15.81360314 +0000 UTC m=+759.481643878" Nov 03 08:52:16.672759 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:16.672731 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:52:19.033873 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:19.033297 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:19.736193 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:19.735191 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:19.930758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:19.930720 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:21.203934 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:21.203661 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:21.723221 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:21.723192 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:52:24.098426 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.098393 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:24.279075 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.279048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:24.415162 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.415108 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:24.574474 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.574444 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:24.574605 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.574498 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:24.773435 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.773368 4543 generic.go:356] "Generic (PLEG): container finished" podID="57713756-4173-4d37-b2e3-7d914e603247" containerID="81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25" exitCode=0 Nov 03 08:52:24.773574 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.773434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerDied","Data":"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25"} Nov 03 08:52:24.773851 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.773831 4543 scope.go:122] "RemoveContainer" containerID="81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25" Nov 03 08:52:24.775200 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775172 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:24.775298 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775236 4543 generic.go:356] "Generic (PLEG): container finished" podID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerID="0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7" exitCode=0 Nov 03 08:52:24.775298 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775253 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:24.775298 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerDied","Data":"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7"} Nov 03 08:52:24.775456 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775304 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:24.775574 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.775557 4543 scope.go:122] "RemoveContainer" containerID="0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7" Nov 03 08:52:24.776773 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.776746 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:24.776855 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.776805 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:24.776855 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:24.776835 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:25.577620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:25.577590 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:52:25.821621 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:25.821577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerStarted","Data":"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5"} Nov 03 08:52:25.832496 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:25.830574 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerStarted","Data":"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f"} Nov 03 08:52:25.934529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:25.934476 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:26.058692 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.058659 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:52:26.059640 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.059617 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 08:52:26.059751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.059661 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 08:52:26.131167 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.122765 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:52:26.131167 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.122917 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:26.131167 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:26.122983 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:27.058920 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:27.058876 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 08:52:27.059345 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:27.058939 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 08:52:27.123856 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:27.123823 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:27.124181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:27.124126 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:28.059394 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:28.059362 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 08:52:28.059777 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:28.059410 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 08:52:28.118568 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:28.118531 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:28.119388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:28.119358 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:29.059062 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.058768 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 08:52:29.059062 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.058812 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 08:52:29.120604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.120480 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:29.120604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.120540 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:29.344720 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.344617 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:29.799165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:29.798358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:30.059346 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:30.059268 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 08:52:30.059346 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:30.059321 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 08:52:30.118553 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:30.118519 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:52:30.118848 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:30.118819 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:52:31.062651 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.062621 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:52:31.063091 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.062964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:52:31.063586 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.063567 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:52:31.122248 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.122229 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:52:31.122516 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.122490 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:52:31.123113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:31.123093 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:52:34.092427 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.092391 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 08:52:34.095582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.095566 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.116628 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.116600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.116729 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.116641 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r4l\" (UniqueName: \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.116768 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.116726 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.116768 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.116749 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.201631 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.201592 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-98d5dd-serving-cert\"" Nov 03 08:52:34.203989 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.203966 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 08:52:34.217582 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.217552 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.217698 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.217601 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r4l\" (UniqueName: \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.217698 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.217654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.217698 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.217680 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.217851 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:34.217842 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-serving-cert: secret "maas-api-e2e-worker-w3-98d5dd-serving-cert" not found Nov 03 08:52:34.217920 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:34.217909 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls podName:40b95d18-8494-4c60-a900-a7eeb091e82d nodeName:}" failed. No retries permitted until 2026-11-03 08:52:34.717889802 +0000 UTC m=+778.385930534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls") pod "maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" (UID: "40b95d18-8494-4c60-a900-a7eeb091e82d") : secret "maas-api-e2e-worker-w3-98d5dd-serving-cert" not found Nov 03 08:52:34.218296 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.218278 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.220066 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.220049 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.277042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.277016 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r4l\" (UniqueName: \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.283993 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.283964 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:34.449041 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.448994 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:34.593533 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.593501 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:34.723265 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.723180 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:34.725632 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:34.725613 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:35.007098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:35.007016 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:35.381980 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:35.381958 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 08:52:35.383811 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:52:35.383784 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b95d18_8494_4c60_a900_a7eeb091e82d.slice/crio-398487228fdec36b23bea0cdb1d9e4c1c1707584d8d50431af8ce3e61ad4fbc8 WatchSource:0}: Error finding container 398487228fdec36b23bea0cdb1d9e4c1c1707584d8d50431af8ce3e61ad4fbc8: Status 404 returned error can't find the container with id 398487228fdec36b23bea0cdb1d9e4c1c1707584d8d50431af8ce3e61ad4fbc8 Nov 03 08:52:35.559732 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:52:35.559696 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:52:35.891102 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:35.891071 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" event={"ID":"40b95d18-8494-4c60-a900-a7eeb091e82d","Type":"ContainerStarted","Data":"398487228fdec36b23bea0cdb1d9e4c1c1707584d8d50431af8ce3e61ad4fbc8"} Nov 03 08:52:36.898453 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:36.898418 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" event={"ID":"40b95d18-8494-4c60-a900-a7eeb091e82d","Type":"ContainerStarted","Data":"6153e495b3d7f3697a9fc1b9e08b88d96258bfdccaebbb2bde028c5749685ab4"} Nov 03 08:52:36.898885 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:36.898553 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:37.017687 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:37.017637 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" podStartSLOduration=2.19913066 podStartE2EDuration="3.017623238s" totalImagesPullingTime="818.492578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:37.015909961 +0000 UTC m=+780.683950699" watchObservedRunningTime="2026-11-03 08:52:37.017623238 +0000 UTC m=+780.685663974" Nov 03 08:52:39.845845 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:39.845810 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:42.908057 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:42.908025 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 08:52:45.004402 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:45.004369 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:50.392399 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:50.392360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:50.670406 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:50.670314 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:52:50.670406 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:50.670387 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:52:52.998110 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.998074 4543 generic.go:356] "Generic (PLEG): container finished" podID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerID="d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611" exitCode=0 Nov 03 08:52:52.998541 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.998171 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerDied","Data":"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611"} Nov 03 08:52:52.998541 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.998525 4543 scope.go:122] "RemoveContainer" containerID="d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611" Nov 03 08:52:52.999811 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.999781 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:52.999911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.999855 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:52.999911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:52.999886 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:52:54.013324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:54.013294 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerStarted","Data":"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca"} Nov 03 08:52:54.307412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:54.307320 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:54.307676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:54.307655 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:52:54.307717 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:54.307702 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:52:55.307778 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:55.307743 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:52:55.308318 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:55.307805 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:52:55.877265 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:55.877236 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:56.307326 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:56.307238 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:52:56.307326 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:56.307293 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:52:57.307716 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:57.307678 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:52:57.308111 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:57.307737 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:52:58.308007 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:58.307965 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 08:52:58.308485 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:58.308028 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 08:52:59.310785 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:59.310754 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:59.311247 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:59.311172 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:59.311832 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:59.311815 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 08:52:59.445549 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:59.445518 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:52:59.587402 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:52:59.587322 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:00.630528 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.630489 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:53:00.631263 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.630938 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" podUID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" containerName="maas-api" containerID="cri-o://7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3" gracePeriod=30 Nov 03 08:53:00.727422 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.727376 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:00.883822 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.883770 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:53:00.999920 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.999890 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\") pod \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " Nov 03 08:53:01.000109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:00.999968 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\" (UniqueName: \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\") pod \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " Nov 03 08:53:01.000109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.000026 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\") pod \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " Nov 03 08:53:01.000109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.000081 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") pod \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\" (UID: \"f541447b-2668-42b5-b3ed-1d1faa1ee02f\") " Nov 03 08:53:01.000462 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.000437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca" pod "f541447b-2668-42b5-b3ed-1d1faa1ee02f" (UID: "f541447b-2668-42b5-b3ed-1d1faa1ee02f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:01.000565 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.000518 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f541447b-2668-42b5-b3ed-1d1faa1ee02f-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:01.002184 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.002155 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs" pod "f541447b-2668-42b5-b3ed-1d1faa1ee02f" (UID: "f541447b-2668-42b5-b3ed-1d1faa1ee02f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:01.002314 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.002239 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px" pod "f541447b-2668-42b5-b3ed-1d1faa1ee02f" (UID: "f541447b-2668-42b5-b3ed-1d1faa1ee02f"). InnerVolumeSpecName "kube-api-access-v57px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:01.002314 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.002267 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls" pod "f541447b-2668-42b5-b3ed-1d1faa1ee02f" (UID: "f541447b-2668-42b5-b3ed-1d1faa1ee02f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:01.050596 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.050560 4543 generic.go:356] "Generic (PLEG): container finished" podID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" containerID="7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3" exitCode=0 Nov 03 08:53:01.050766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.050602 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" Nov 03 08:53:01.050766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.050620 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" event={"ID":"f541447b-2668-42b5-b3ed-1d1faa1ee02f","Type":"ContainerDied","Data":"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3"} Nov 03 08:53:01.050766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.050682 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6" event={"ID":"f541447b-2668-42b5-b3ed-1d1faa1ee02f","Type":"ContainerDied","Data":"2a23db3fcbdfc44e2f27045aeb7240c6d5bfcf3a2333fa5c92f77a814a81131c"} Nov 03 08:53:01.050766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.050712 4543 scope.go:122] "RemoveContainer" containerID="7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3" Nov 03 08:53:01.060473 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.060456 4543 scope.go:122] "RemoveContainer" containerID="7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3" Nov 03 08:53:01.060746 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:01.060726 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3\": container with ID starting with 7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3 not found: ID does not exist" containerID="7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3" Nov 03 08:53:01.060791 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.060753 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3"} err="failed to get container status \"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3\": rpc error: code = NotFound desc = could not find container \"7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3\": container with ID starting with 7d89c0d66a9f4c136c5424601a7006034240d3d3e4465d357ab18dce477b85e3 not found: ID does not exist" Nov 03 08:53:01.102047 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.102021 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57px\" (UniqueName: \"kubernetes.io/projected/f541447b-2668-42b5-b3ed-1d1faa1ee02f-kube-api-access-v57px\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:01.102047 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.102049 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:01.102310 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.102064 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f541447b-2668-42b5-b3ed-1d1faa1ee02f-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:01.221321 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.221241 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:53:01.226673 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.226648 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82eec5e9-686b74bf84-p5jh6"] Nov 03 08:53:01.368847 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.368791 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:01.504241 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:01.504124 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:02.450640 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:02.450607 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" path="/var/lib/kubelet/pods/f541447b-2668-42b5-b3ed-1d1faa1ee02f/volumes" Nov 03 08:53:02.757297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:02.757212 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:04.574204 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:04.574171 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:04.707953 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:04.707922 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:05.560282 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:05.560251 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:53:07.571252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.571223 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:53:07.571681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.571433 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" podUID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" containerName="maas-api" containerID="cri-o://12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c" gracePeriod=30 Nov 03 08:53:07.662652 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.662607 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:07.844416 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.844386 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:53:07.974367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.974340 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") pod \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " Nov 03 08:53:07.974560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.974446 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\") pod \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " Nov 03 08:53:07.974560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.974496 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\" (UniqueName: \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\") pod \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " Nov 03 08:53:07.974560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.974555 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\") pod \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\" (UID: \"e62b8353-aaef-4a9d-a6d3-ffc6b200664a\") " Nov 03 08:53:07.974841 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.974809 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca" pod "e62b8353-aaef-4a9d-a6d3-ffc6b200664a" (UID: "e62b8353-aaef-4a9d-a6d3-ffc6b200664a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:07.976838 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.976812 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs" pod "e62b8353-aaef-4a9d-a6d3-ffc6b200664a" (UID: "e62b8353-aaef-4a9d-a6d3-ffc6b200664a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:07.977068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.977053 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls" pod "e62b8353-aaef-4a9d-a6d3-ffc6b200664a" (UID: "e62b8353-aaef-4a9d-a6d3-ffc6b200664a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:07.977185 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:07.977120 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf" pod "e62b8353-aaef-4a9d-a6d3-ffc6b200664a" (UID: "e62b8353-aaef-4a9d-a6d3-ffc6b200664a"). InnerVolumeSpecName "kube-api-access-v9kdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:08.090448 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.090369 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:08.090448 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.090400 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9kdf\" (UniqueName: \"kubernetes.io/projected/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-kube-api-access-v9kdf\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:08.090448 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.090417 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:08.090448 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.090430 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e62b8353-aaef-4a9d-a6d3-ffc6b200664a-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:08.092678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.092646 4543 generic.go:356] "Generic (PLEG): container finished" podID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" containerID="12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c" exitCode=0 Nov 03 08:53:08.092842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.092693 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" Nov 03 08:53:08.092842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.092726 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" event={"ID":"e62b8353-aaef-4a9d-a6d3-ffc6b200664a","Type":"ContainerDied","Data":"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c"} Nov 03 08:53:08.092842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.092786 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb" event={"ID":"e62b8353-aaef-4a9d-a6d3-ffc6b200664a","Type":"ContainerDied","Data":"f8f756f9993d1b54db6da3179fa5aac5b95b541d39d6bd8b4583802cdef8b7de"} Nov 03 08:53:08.092842 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.092828 4543 scope.go:122] "RemoveContainer" containerID="12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c" Nov 03 08:53:08.102747 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.102728 4543 scope.go:122] "RemoveContainer" containerID="12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c" Nov 03 08:53:08.103001 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:08.102982 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c\": container with ID starting with 12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c not found: ID does not exist" containerID="12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c" Nov 03 08:53:08.103082 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.103010 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c"} err="failed to get container status \"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c\": rpc error: code = NotFound desc = could not find container \"12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c\": container with ID starting with 12de0f46ab777f469d01fc2ccb2e915d87fb56a97262a80ce5e86e28edf59d8c not found: ID does not exist" Nov 03 08:53:08.261256 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.261228 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:53:08.266368 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.266346 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6a1442a9-5d7f7b965d-hhlzb"] Nov 03 08:53:08.449967 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.449937 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" path="/var/lib/kubelet/pods/e62b8353-aaef-4a9d-a6d3-ffc6b200664a/volumes" Nov 03 08:53:08.857824 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.857739 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:53:08.858303 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.857984 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" podUID="950f126c-5642-4822-b947-35b7b1838c4d" containerName="maas-api" containerID="cri-o://23aeddb0cf409d9a644ba8248a1e766c9c50d35abf5d63bee6f1c1361f3d17c9" gracePeriod=30 Nov 03 08:53:08.953362 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:08.953330 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:09.088205 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.088168 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:09.099652 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.099626 4543 generic.go:356] "Generic (PLEG): container finished" podID="950f126c-5642-4822-b947-35b7b1838c4d" containerID="23aeddb0cf409d9a644ba8248a1e766c9c50d35abf5d63bee6f1c1361f3d17c9" exitCode=0 Nov 03 08:53:09.100059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.099695 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" event={"ID":"950f126c-5642-4822-b947-35b7b1838c4d","Type":"ContainerDied","Data":"23aeddb0cf409d9a644ba8248a1e766c9c50d35abf5d63bee6f1c1361f3d17c9"} Nov 03 08:53:09.100059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.099750 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" event={"ID":"950f126c-5642-4822-b947-35b7b1838c4d","Type":"ContainerDied","Data":"0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644"} Nov 03 08:53:09.100059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.099773 4543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dd54472522f41846cfeaa000fc70cb6a621eff8b3332e9fe9f9b4c58ad39644" Nov 03 08:53:09.112870 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.112852 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:53:09.304941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.304912 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") pod \"950f126c-5642-4822-b947-35b7b1838c4d\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " Nov 03 08:53:09.305115 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.304976 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\") pod \"950f126c-5642-4822-b947-35b7b1838c4d\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " Nov 03 08:53:09.305115 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.305022 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\") pod \"950f126c-5642-4822-b947-35b7b1838c4d\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " Nov 03 08:53:09.305281 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.305194 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\" (UniqueName: \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\") pod \"950f126c-5642-4822-b947-35b7b1838c4d\" (UID: \"950f126c-5642-4822-b947-35b7b1838c4d\") " Nov 03 08:53:09.305463 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.305433 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca" pod "950f126c-5642-4822-b947-35b7b1838c4d" (UID: "950f126c-5642-4822-b947-35b7b1838c4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:09.305605 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.305587 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/950f126c-5642-4822-b947-35b7b1838c4d-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.307269 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.307239 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw" pod "950f126c-5642-4822-b947-35b7b1838c4d" (UID: "950f126c-5642-4822-b947-35b7b1838c4d"). InnerVolumeSpecName "kube-api-access-jmmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:09.307357 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.307315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs" pod "950f126c-5642-4822-b947-35b7b1838c4d" (UID: "950f126c-5642-4822-b947-35b7b1838c4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:09.307357 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.307324 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls" pod "950f126c-5642-4822-b947-35b7b1838c4d" (UID: "950f126c-5642-4822-b947-35b7b1838c4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 08:53:09.406545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.406485 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmjw\" (UniqueName: \"kubernetes.io/projected/950f126c-5642-4822-b947-35b7b1838c4d-kube-api-access-jmmjw\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.406545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.406507 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.406545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.406516 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950f126c-5642-4822-b947-35b7b1838c4d-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:09.419275 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:09.419251 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:10.106300 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.106268 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp" Nov 03 08:53:10.186999 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.186970 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:10.285990 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.285966 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:53:10.336445 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.336419 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b6ef5ff5-8659b56c8c-gx8zp"] Nov 03 08:53:10.450294 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.450266 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950f126c-5642-4822-b947-35b7b1838c4d" path="/var/lib/kubelet/pods/950f126c-5642-4822-b947-35b7b1838c4d/volumes" Nov 03 08:53:10.499182 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.499133 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:10.928898 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:10.928859 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:11.097120 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.097084 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:11.099020 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.098999 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:53:11.099229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.099209 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" containerID="cri-o://c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5" gracePeriod=30 Nov 03 08:53:11.300837 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.300746 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:53:11.301294 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.301062 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" containerID="cri-o://7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f" gracePeriod=30 Nov 03 08:53:11.338686 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.338660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:11.467619 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.467587 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:11.667098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.667070 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:11.852764 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:11.852738 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.015072 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.015002 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.146324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.146289 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.422583 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.422539 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.562953 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.562910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.706543 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.706465 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.711715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.711691 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 08:53:12.712415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712395 4543 cpu_manager.go:414] "containerMap: removing container" podUID="950f126c-5642-4822-b947-35b7b1838c4d" containerName="maas-api" Nov 03 08:53:12.712415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712416 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950f126c-5642-4822-b947-35b7b1838c4d" containerName="maas-api" Nov 03 08:53:12.712572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712430 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" containerName="maas-api" Nov 03 08:53:12.712572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712436 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" containerName="maas-api" Nov 03 08:53:12.712572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712473 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" containerName="maas-api" Nov 03 08:53:12.712572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712501 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" containerName="maas-api" Nov 03 08:53:12.712699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712599 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62b8353-aaef-4a9d-a6d3-ffc6b200664a" containerName="maas-api" Nov 03 08:53:12.712699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712614 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="950f126c-5642-4822-b947-35b7b1838c4d" containerName="maas-api" Nov 03 08:53:12.712699 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.712625 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f541447b-2668-42b5-b3ed-1d1faa1ee02f" containerName="maas-api" Nov 03 08:53:12.715994 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.715976 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739160 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739118 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqgq\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739205 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739239 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739267 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739259 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739396 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739363 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739438 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739392 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739438 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739420 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739564 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739545 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.739618 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.739596 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.796181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.796134 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 08:53:12.829894 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.829870 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\"" Nov 03 08:53:12.837928 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.837902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:12.840230 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840214 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840244 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840293 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqgq\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840322 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840315 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840472 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840338 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840472 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840366 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840472 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840472 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840665 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840486 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840693 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.840802 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.840779 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.841079 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.841033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.841079 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.841105 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.841296 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.841260 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.842527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.842508 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.842829 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.842814 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.904097 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.904066 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.904097 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.904089 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqgq\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:12.985750 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:12.985681 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:13.028361 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.028332 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:13.114663 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.114634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:13.457502 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.457477 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 08:53:13.459851 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:53:13.459809 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5717801_8479_4b75_8f4f_4ee3b9c78f94.slice/crio-51102261032a0b352493351bdebc31fe28d1a1b60eb4df0d2caea655ac004475 WatchSource:0}: Error finding container 51102261032a0b352493351bdebc31fe28d1a1b60eb4df0d2caea655ac004475: Status 404 returned error can't find the container with id 51102261032a0b352493351bdebc31fe28d1a1b60eb4df0d2caea655ac004475 Nov 03 08:53:13.462998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.462961 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:13.463113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.463050 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:13.463113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.463090 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:13.750122 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:13.750029 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:14.001123 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:14.001044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:14.129505 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:14.129476 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerStarted","Data":"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b"} Nov 03 08:53:14.129656 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:14.129523 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerStarted","Data":"51102261032a0b352493351bdebc31fe28d1a1b60eb4df0d2caea655ac004475"} Nov 03 08:53:14.249706 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:14.249653 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podStartSLOduration=2.249639456 podStartE2EDuration="2.249639456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:14.248887913 +0000 UTC m=+817.916928653" watchObservedRunningTime="2026-11-03 08:53:14.249639456 +0000 UTC m=+817.917680193" Nov 03 08:53:15.028998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:15.028965 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:15.034422 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:15.034397 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:15.135417 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:15.135393 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:15.136127 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:15.136112 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:53:15.593390 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:15.593357 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:16.065746 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.065716 4543 patch_prober.go:29] interesting pod/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 08:53:16.066170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.065765 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 08:53:16.125564 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.125537 4543 patch_prober.go:29] interesting pod/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 08:53:16.125745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.125719 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 08:53:16.197283 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.197254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:16.327227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.327194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:16.429115 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.429089 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:53:16.472938 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.472907 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.472959 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.472988 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473025 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473053 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473080 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473161 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473231 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473288 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\") pod \"57713756-4173-4d37-b2e3-7d914e603247\" (UID: \"57713756-4173-4d37-b2e3-7d914e603247\") " Nov 03 08:53:16.473586 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.473727 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473659 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.473805 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473780 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.473805 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.473790 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.476358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.476040 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.476358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.476330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:16.476993 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.476878 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:53:16.476993 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.476958 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.479545 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.479520 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:16.479623 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.479568 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm" pod "57713756-4173-4d37-b2e3-7d914e603247" (UID: "57713756-4173-4d37-b2e3-7d914e603247"). InnerVolumeSpecName "kube-api-access-5g9zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:16.539817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.539794 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:53:16.573970 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.573912 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.573970 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.573962 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.573982 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574019 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574086 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574103 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574213 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574257 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\") pod \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\" (UID: \"2742b2f3-a7c0-4428-8d0a-589e67781fe5\") " Nov 03 08:53:16.574324 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574275 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:53:16.574464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574361 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.574464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574452 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.574604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574581 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.574604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574595 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.574762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574745 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g9zm\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-kube-api-access-5g9zm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.574817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574769 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.574817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574783 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57713756-4173-4d37-b2e3-7d914e603247-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.574817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574796 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.574817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574809 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57713756-4173-4d37-b2e3-7d914e603247-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574822 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574835 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574848 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57713756-4173-4d37-b2e3-7d914e603247-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574860 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574874 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.574899 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57713756-4173-4d37-b2e3-7d914e603247-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.575000 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.575214 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.575069 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.576416 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.576391 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:16.576520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.576413 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "kube-api-access-pf4pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:53:16.576520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.576423 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:53:16.576601 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.576549 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo" pod "2742b2f3-a7c0-4428-8d0a-589e67781fe5" (UID: "2742b2f3-a7c0-4428-8d0a-589e67781fe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:53:16.676630 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.676582 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.676630 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.676631 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4pp\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-kube-api-access-pf4pp\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.676823 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.676648 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:16.676823 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:16.676663 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2742b2f3-a7c0-4428-8d0a-589e67781fe5-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:53:17.152204 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.152171 4543 generic.go:356] "Generic (PLEG): container finished" podID="57713756-4173-4d37-b2e3-7d914e603247" containerID="c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5" exitCode=0 Nov 03 08:53:17.152615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.152233 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" Nov 03 08:53:17.152615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.152254 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerDied","Data":"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5"} Nov 03 08:53:17.152615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.152334 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7" event={"ID":"57713756-4173-4d37-b2e3-7d914e603247","Type":"ContainerDied","Data":"b7a249cdb149ee609dc084136ed1b22172f3a046ecbde23eaf930b4539916b37"} Nov 03 08:53:17.152615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.152362 4543 scope.go:122] "RemoveContainer" containerID="c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5" Nov 03 08:53:17.154334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.154299 4543 generic.go:356] "Generic (PLEG): container finished" podID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerID="7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f" exitCode=0 Nov 03 08:53:17.154440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.154355 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" Nov 03 08:53:17.154440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.154377 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerDied","Data":"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f"} Nov 03 08:53:17.154440 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.154424 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9" event={"ID":"2742b2f3-a7c0-4428-8d0a-589e67781fe5","Type":"ContainerDied","Data":"0a718003b4dd7adab6d42cb59ed538862a1a2e8a6065c6fba85f690df50e76da"} Nov 03 08:53:17.162778 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.162759 4543 scope.go:122] "RemoveContainer" containerID="81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25" Nov 03 08:53:17.170840 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.170824 4543 scope.go:122] "RemoveContainer" containerID="c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5" Nov 03 08:53:17.171094 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:17.171077 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5\": container with ID starting with c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5 not found: ID does not exist" containerID="c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5" Nov 03 08:53:17.171177 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.171101 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5"} err="failed to get container status \"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5\": rpc error: code = NotFound desc = could not find container \"c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5\": container with ID starting with c3d7c71568104a65c583b404f2cd4c5dc47635ffc04818b3b9cbd4a5449f95b5 not found: ID does not exist" Nov 03 08:53:17.171177 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.171118 4543 scope.go:122] "RemoveContainer" containerID="81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25" Nov 03 08:53:17.171383 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:17.171367 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25\": container with ID starting with 81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25 not found: ID does not exist" containerID="81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25" Nov 03 08:53:17.171447 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.171393 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25"} err="failed to get container status \"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25\": rpc error: code = NotFound desc = could not find container \"81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25\": container with ID starting with 81f119c65efafb93fd93e5947a50ffe1491ccec0267473cf236f5bcb21e04a25 not found: ID does not exist" Nov 03 08:53:17.171447 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.171416 4543 scope.go:122] "RemoveContainer" containerID="7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f" Nov 03 08:53:17.179045 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.179030 4543 scope.go:122] "RemoveContainer" containerID="0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7" Nov 03 08:53:17.186590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.186570 4543 scope.go:122] "RemoveContainer" containerID="7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f" Nov 03 08:53:17.186914 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:17.186882 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f\": container with ID starting with 7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f not found: ID does not exist" containerID="7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f" Nov 03 08:53:17.186997 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.186921 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f"} err="failed to get container status \"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f\": rpc error: code = NotFound desc = could not find container \"7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f\": container with ID starting with 7c4a14ebbeb40a0dabd7f6c3f22998f0257ad37930e611ee67bae254e0f3d26f not found: ID does not exist" Nov 03 08:53:17.186997 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.186939 4543 scope.go:122] "RemoveContainer" containerID="0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7" Nov 03 08:53:17.187242 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:17.187213 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7\": container with ID starting with 0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7 not found: ID does not exist" containerID="0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7" Nov 03 08:53:17.187296 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.187240 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7"} err="failed to get container status \"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7\": rpc error: code = NotFound desc = could not find container \"0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7\": container with ID starting with 0bbffd94b13362fafadcb21216056b1844efa157087b59c8820032be6d27ecb7 not found: ID does not exist" Nov 03 08:53:17.322585 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.322554 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:53:17.373801 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.373730 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6a1442a9-openshift-default-7469b7cb49-kznc9"] Nov 03 08:53:17.409538 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.409504 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:17.486769 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.486736 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:53:17.492647 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.492619 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-82eec5e9-openshift-default-777bf698b9-5rjn7"] Nov 03 08:53:17.593181 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.593123 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:17.735572 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.735484 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:17.878536 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:17.878501 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:18.321401 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:18.321365 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:18.450513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:18.450482 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" path="/var/lib/kubelet/pods/2742b2f3-a7c0-4428-8d0a-589e67781fe5/volumes" Nov 03 08:53:18.450958 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:18.450944 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57713756-4173-4d37-b2e3-7d914e603247" path="/var/lib/kubelet/pods/57713756-4173-4d37-b2e3-7d914e603247/volumes" Nov 03 08:53:18.499903 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:18.499862 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:18.664355 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:18.664323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:21.154928 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:21.154864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:21.890292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:21.890261 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:22.565888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:22.565849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:23.056709 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.056678 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 08:53:23.057299 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057281 4543 cpu_manager.go:414] "containerMap: removing container" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:23.057358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057300 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:23.057358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057308 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:23.057358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057315 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:23.057358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057343 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:23.057358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057349 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:23.057514 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057424 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:23.057514 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057431 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:23.057514 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.057439 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:23.060558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.060541 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.143298 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.143267 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 08:53:23.146390 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146359 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146541 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146402 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146541 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146432 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2gl\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146541 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146459 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146625 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146705 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146796 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146730 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146837 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146805 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.146944 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.146921 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.147223 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.147202 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:23.169101 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.169082 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26e861-openshift-default-dockercfg-4wl7s\"" Nov 03 08:53:23.248772 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.248723 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249045 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249024 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249176 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249162 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249366 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249335 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249701 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249685 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249799 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249786 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2gl\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.249895 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.250018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.250005 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.250728 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.250698 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.250852 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.250766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.251122 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.251099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.251630 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.251606 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.251930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.249578 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.258719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.257651 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.258909 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.258780 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.281248 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.281219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:23.310968 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.310889 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.311079 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.311003 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2gl\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.377868 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.377824 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:23.422698 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.422660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:23.809653 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.809622 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 08:53:23.845539 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:53:23.845509 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a93cd66_8b94_4ba0_bbc3_448d61c19de3.slice/crio-23d1e1950808eb504aa2c9287398859dcea93728fbd68f87c17e7dbc433dbaad WatchSource:0}: Error finding container 23d1e1950808eb504aa2c9287398859dcea93728fbd68f87c17e7dbc433dbaad: Status 404 returned error can't find the container with id 23d1e1950808eb504aa2c9287398859dcea93728fbd68f87c17e7dbc433dbaad Nov 03 08:53:23.847644 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.847613 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:23.847752 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.847694 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:23.847752 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:23.847736 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:53:24.181460 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.181405 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:24.196559 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.196525 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerStarted","Data":"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1"} Nov 03 08:53:24.196741 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.196586 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerStarted","Data":"23d1e1950808eb504aa2c9287398859dcea93728fbd68f87c17e7dbc433dbaad"} Nov 03 08:53:24.367914 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.367879 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:24.378995 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.378975 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:24.384525 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.384502 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:24.507118 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:24.507027 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podStartSLOduration=1.507012453 podStartE2EDuration="1.507012453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:24.31937833 +0000 UTC m=+827.987419069" watchObservedRunningTime="2026-11-03 08:53:24.507012453 +0000 UTC m=+828.175053268" Nov 03 08:53:25.201548 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:25.201518 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:25.202564 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:25.202546 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:53:27.057995 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:27.057961 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:27.282822 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:27.282787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:27.417908 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:27.417864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:28.389598 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:28.389559 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:28.623532 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:28.623500 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:28.832056 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:28.831982 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:30.051150 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:30.051108 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:30.208190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:30.208161 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:30.344686 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:30.344604 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:31.514971 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:31.514933 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:31.697825 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:31.697792 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:33.808760 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.808731 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 08:53:33.809172 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.809159 4543 cpu_manager.go:414] "containerMap: removing container" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:33.809220 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.809173 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57713756-4173-4d37-b2e3-7d914e603247" containerName="istio-proxy" Nov 03 08:53:33.809273 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.809264 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2742b2f3-a7c0-4428-8d0a-589e67781fe5" containerName="istio-proxy" Nov 03 08:53:33.812558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.812543 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:33.925354 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.925317 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-77207e-serving-cert\"" Nov 03 08:53:33.927510 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.927487 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 08:53:33.960473 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.960447 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:33.960625 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.960599 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:33.960689 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.960657 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:33.960775 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:33.960754 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6l2\" (UniqueName: \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.047563 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.047528 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:34.061655 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.061592 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6l2\" (UniqueName: \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.061771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.061687 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.061771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.061734 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.061839 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.061768 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.061839 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:34.061826 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-serving-cert: secret "maas-api-e2e-worker-w2-77207e-serving-cert" not found Nov 03 08:53:34.061918 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:34.061907 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls podName:af4de6fd-068d-42dc-b681-9ab65b0d5449 nodeName:}" failed. No retries permitted until 2026-11-03 08:53:34.561888355 +0000 UTC m=+838.229929079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls") pod "maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" (UID: "af4de6fd-068d-42dc-b681-9ab65b0d5449") : secret "maas-api-e2e-worker-w2-77207e-serving-cert" not found Nov 03 08:53:34.062553 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.062534 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.064203 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.064180 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.124599 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.124554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6l2\" (UniqueName: \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.181068 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.181040 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:34.310851 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.310809 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:34.566612 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.566583 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.569107 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.569079 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") pod \"maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:34.730059 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:34.730026 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:35.141860 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:35.141837 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 08:53:35.143664 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:53:35.143627 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4de6fd_068d_42dc_b681_9ab65b0d5449.slice/crio-f96f8dff1ea861ad13c8d8c98662b7a85fd3f158d6c908e4bee90ab8d0e35462 WatchSource:0}: Error finding container f96f8dff1ea861ad13c8d8c98662b7a85fd3f158d6c908e4bee90ab8d0e35462: Status 404 returned error can't find the container with id f96f8dff1ea861ad13c8d8c98662b7a85fd3f158d6c908e4bee90ab8d0e35462 Nov 03 08:53:35.253102 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:35.253073 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" event={"ID":"af4de6fd-068d-42dc-b681-9ab65b0d5449","Type":"ContainerStarted","Data":"f96f8dff1ea861ad13c8d8c98662b7a85fd3f158d6c908e4bee90ab8d0e35462"} Nov 03 08:53:35.559991 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:35.559914 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:53:36.259629 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:36.259592 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" event={"ID":"af4de6fd-068d-42dc-b681-9ab65b0d5449","Type":"ContainerStarted","Data":"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2"} Nov 03 08:53:36.260001 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:36.259684 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:36.378412 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:36.378357 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" podStartSLOduration=2.503088995 podStartE2EDuration="3.378339317s" totalImagesPullingTime="875.250322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:36.377659144 +0000 UTC m=+840.045699911" watchObservedRunningTime="2026-11-03 08:53:36.378339317 +0000 UTC m=+840.046380056" Nov 03 08:53:36.479861 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:36.479833 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:36.666347 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:36.666279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:39.239345 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:39.239313 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:42.269841 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:42.269814 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 08:53:44.262733 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:44.262692 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:48.467843 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:48.467764 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:49.101937 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.101905 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 08:53:49.107032 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.107011 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.114672 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.114641 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.114775 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.114684 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.114775 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.114770 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4962p\" (UniqueName: \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.114870 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.114816 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.211579 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.211555 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-26e861-serving-cert\"" Nov 03 08:53:49.213907 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.213887 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 08:53:49.216281 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.216263 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4962p\" (UniqueName: \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.216351 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.216297 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.216391 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.216369 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.216437 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.216391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.216508 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:49.216486 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-serving-cert: secret "maas-api-e2e-worker-w1-26e861-serving-cert" not found Nov 03 08:53:49.216613 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:49.216597 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls podName:14c04f07-2576-478b-a56a-c12c3e322119 nodeName:}" failed. No retries permitted until 2026-11-03 08:53:49.716567912 +0000 UTC m=+853.384608628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls") pod "maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" (UID: "14c04f07-2576-478b-a56a-c12c3e322119") : secret "maas-api-e2e-worker-w1-26e861-serving-cert" not found Nov 03 08:53:49.217000 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.216980 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.218821 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.218805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.274906 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.274872 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4962p\" (UniqueName: \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.290565 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.290539 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:49.453832 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.453805 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:49.588165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.588103 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:49.721651 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.721574 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:49.723965 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:49.723943 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") pod \"maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:50.020178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:50.020084 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:50.391560 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:50.391536 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 08:53:50.392721 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:53:50.392689 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c04f07_2576_478b_a56a_c12c3e322119.slice/crio-a4dc12c489819451005baa415c57935411dcb36f8a02957b27a7778213420637 WatchSource:0}: Error finding container a4dc12c489819451005baa415c57935411dcb36f8a02957b27a7778213420637: Status 404 returned error can't find the container with id a4dc12c489819451005baa415c57935411dcb36f8a02957b27a7778213420637 Nov 03 08:53:51.335820 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:51.335787 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" event={"ID":"14c04f07-2576-478b-a56a-c12c3e322119","Type":"ContainerStarted","Data":"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4"} Nov 03 08:53:51.336238 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:51.335852 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" event={"ID":"14c04f07-2576-478b-a56a-c12c3e322119","Type":"ContainerStarted","Data":"a4dc12c489819451005baa415c57935411dcb36f8a02957b27a7778213420637"} Nov 03 08:53:51.336238 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:51.335881 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:51.454553 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:51.454461 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" podStartSLOduration=1.67909211 podStartE2EDuration="2.454447913s" totalImagesPullingTime="775.355803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:51.452751708 +0000 UTC m=+855.120792444" watchObservedRunningTime="2026-11-03 08:53:51.454447913 +0000 UTC m=+855.122488673" Nov 03 08:53:54.320483 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.320453 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 08:53:54.324234 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.324217 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.382327 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.382294 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.382469 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.382342 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdg7h\" (UniqueName: \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.382469 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.382466 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.382587 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.382495 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.430048 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.430019 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-910d50-serving-cert\"" Nov 03 08:53:54.432275 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.432253 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 08:53:54.483633 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.483586 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.483817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.483645 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdg7h\" (UniqueName: \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.483817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.483722 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.483817 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:54.483727 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-serving-cert: secret "maas-api-e2e-worker-w0-910d50-serving-cert" not found Nov 03 08:53:54.483817 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.483755 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.483817 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:53:54.483793 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls podName:f77e93d9-113b-41ce-b575-796a92e08e82 nodeName:}" failed. No retries permitted until 2026-11-03 08:53:54.983770766 +0000 UTC m=+858.651811500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls") pod "maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" (UID: "f77e93d9-113b-41ce-b575-796a92e08e82") : secret "maas-api-e2e-worker-w0-910d50-serving-cert" not found Nov 03 08:53:54.484559 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.484539 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.486348 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.486327 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.515156 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.515109 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:54.543020 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.542994 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdg7h\" (UniqueName: \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.676964 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.676935 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:54.807943 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.807903 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:53:54.988113 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.988035 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:54.990567 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:54.990539 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") pod \"maas-api-e2e-worker-w0-910d50-677688d75c-qglcl\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:55.236774 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:55.236743 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:55.615106 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:55.615080 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 08:53:55.617320 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:53:55.617292 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77e93d9_113b_41ce_b575_796a92e08e82.slice/crio-efe9a9e091566b549d7c5e2c3fdf157d791e5faac16f01e7388ff67c69116675 WatchSource:0}: Error finding container efe9a9e091566b549d7c5e2c3fdf157d791e5faac16f01e7388ff67c69116675: Status 404 returned error can't find the container with id efe9a9e091566b549d7c5e2c3fdf157d791e5faac16f01e7388ff67c69116675 Nov 03 08:53:56.363819 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:56.363788 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" event={"ID":"f77e93d9-113b-41ce-b575-796a92e08e82","Type":"ContainerStarted","Data":"efe9a9e091566b549d7c5e2c3fdf157d791e5faac16f01e7388ff67c69116675"} Nov 03 08:53:57.345997 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:57.345966 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 08:53:57.371134 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:57.371103 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" event={"ID":"f77e93d9-113b-41ce-b575-796a92e08e82","Type":"ContainerStarted","Data":"105b25208a3301899740416451ac3e28ab0bf919e24e6c6df7495526b3c25f90"} Nov 03 08:53:57.371307 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:57.371249 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:53:57.595567 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:57.592180 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" podStartSLOduration=2.743858265 podStartE2EDuration="3.592159983s" totalImagesPullingTime="848.301718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:57.587602647 +0000 UTC m=+861.255643385" watchObservedRunningTime="2026-11-03 08:53:57.592159983 +0000 UTC m=+861.260200715" Nov 03 08:53:59.752451 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:53:59.752402 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:03.406358 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:03.406306 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 08:54:04.740304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:04.740269 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:05.560329 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:54:05.560295 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:54:09.452751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.452728 4543 generic.go:356] "Generic (PLEG): container finished" podID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerID="310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1" exitCode=0 Nov 03 08:54:09.453205 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.452810 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerDied","Data":"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1"} Nov 03 08:54:09.453205 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.453116 4543 scope.go:122] "RemoveContainer" containerID="310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1" Nov 03 08:54:09.454319 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.454287 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:09.454390 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.454353 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:09.454390 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:09.454385 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:10.042885 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:10.042846 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:10.206288 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:10.206245 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:54:10.465201 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:10.465168 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerStarted","Data":"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224"} Nov 03 08:54:11.378353 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:11.378314 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:54:11.379396 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:11.379375 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 08:54:11.379465 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:11.379446 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 08:54:12.378859 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:12.378829 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 08:54:12.379301 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:12.378877 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 08:54:13.378881 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:13.378844 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 08:54:13.379269 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:13.378912 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 08:54:14.378907 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:14.378872 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 08:54:14.379333 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:14.378926 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 08:54:14.606757 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:14.606718 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:15.156407 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:15.156371 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:15.379269 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:15.379229 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 03 08:54:15.379786 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:15.379307 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 03 08:54:15.393459 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:15.393399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:16.176309 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.176271 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:54:16.176482 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.176332 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:54:16.382792 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.382770 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:54:16.383195 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.383181 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:54:16.384315 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.384301 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 08:54:16.522998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.522916 4543 generic.go:356] "Generic (PLEG): container finished" podID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerID="f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b" exitCode=0 Nov 03 08:54:16.523178 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.523019 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerDied","Data":"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b"} Nov 03 08:54:16.524172 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.524026 4543 scope.go:122] "RemoveContainer" containerID="f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b" Nov 03 08:54:16.526832 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.526792 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:16.526938 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.526863 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:16.526938 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:16.526895 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:54:17.530092 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:17.530056 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerStarted","Data":"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539"} Nov 03 08:54:18.028683 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:18.028655 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:54:18.028854 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:18.028829 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 08:54:18.028906 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:18.028886 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 08:54:19.029087 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:19.029056 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 08:54:19.029480 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:19.029107 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 08:54:19.863050 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:19.863010 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:20.028731 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:20.028692 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 08:54:20.028912 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:20.028751 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 08:54:20.404005 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:20.403970 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:21.028627 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:21.028589 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 08:54:21.028835 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:21.028657 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 08:54:21.882198 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:21.881248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:22.029677 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:22.029647 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 08:54:22.029886 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:22.029721 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 08:54:23.032865 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:23.032832 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:54:23.033351 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:23.033107 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:54:23.033876 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:23.033858 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 08:54:23.842904 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:23.842869 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:23.985429 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:23.985401 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:24.374860 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:24.374824 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:24.522277 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:24.522248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:24.748190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:24.748080 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:25.376782 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:25.376752 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:27.743188 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:27.743132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:28.295389 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:28.295354 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:28.463061 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:28.463030 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:28.621304 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:28.621275 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:30.484244 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:30.484212 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:30.770851 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:30.770772 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:31.411190 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:31.411161 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:31.968531 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:31.968500 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:32.106906 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:32.106878 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:32.256252 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:32.256179 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:35.560565 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:54:35.560527 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:54:36.231431 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:36.231399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:36.527700 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:36.527627 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:54:36.529487 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:36.529467 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:54:41.238359 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:41.238321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:45.227098 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:45.227067 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:46.263538 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:46.263490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:51.375052 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:51.375023 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:54.370588 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:54.370538 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:54:56.438593 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:54:56.438562 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:00.247343 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.247310 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 08:55:00.253548 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.253521 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.306960 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.306928 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 08:55:00.353909 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.353883 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"openshift-service-ca.crt\"" Nov 03 08:55:00.354762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.354741 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"kube-root-ca.crt\"" Nov 03 08:55:00.357714 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.357692 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"simulator-w0-910d50-kserve-self-signed-certs\"" Nov 03 08:55:00.357909 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.357897 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"default-dockercfg-2bds8\"" Nov 03 08:55:00.399656 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.399618 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:00.405383 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405356 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.405537 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405390 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.405537 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405416 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.405617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405597 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.405669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405641 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.405762 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.405744 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwpgr\" (UniqueName: \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507244 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwpgr\" (UniqueName: \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507355 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507513 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507373 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507656 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507519 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507656 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507609 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507765 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507653 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.507765 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.507740 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.509694 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.509670 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.509980 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.509964 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.568518 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.568493 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwpgr\" (UniqueName: \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\") pod \"simulator-w0-910d50-kserve-6d7d59b498-vqb9q\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:00.866474 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:00.866450 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:01.244889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.244862 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 08:55:01.246079 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:55:01.246044 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51cc7f98_70b1_4077_bcb5_3e19ab21ba78.slice/crio-9106cdffcc8560cc7268bc56e485fd8ebddb783ffd968daaa856b50652dde848 WatchSource:0}: Error finding container 9106cdffcc8560cc7268bc56e485fd8ebddb783ffd968daaa856b50652dde848: Status 404 returned error can't find the container with id 9106cdffcc8560cc7268bc56e485fd8ebddb783ffd968daaa856b50652dde848 Nov 03 08:55:01.467931 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.467898 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:01.757465 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.757436 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" event={"ID":"51cc7f98-70b1-4077-bcb5-3e19ab21ba78","Type":"ContainerStarted","Data":"ef338a02a5b2e24d301243d0abdd0c620b8707c723128bd54eee00e0f867f8e5"} Nov 03 08:55:01.757633 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.757473 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" event={"ID":"51cc7f98-70b1-4077-bcb5-3e19ab21ba78","Type":"ContainerStarted","Data":"9106cdffcc8560cc7268bc56e485fd8ebddb783ffd968daaa856b50652dde848"} Nov 03 08:55:01.866924 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.866892 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:01.876297 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:01.876248 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podStartSLOduration=1.876234518 podStartE2EDuration="1.876234518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:55:01.874844656 +0000 UTC m=+925.542885397" watchObservedRunningTime="2026-11-03 08:55:01.876234518 +0000 UTC m=+925.544275256" Nov 03 08:55:05.560329 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:55:05.560292 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:55:06.559968 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:06.559930 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:10.867533 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:10.867503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:10.879534 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:10.879512 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:11.651712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:11.651680 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:11.824503 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:11.824479 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 08:55:14.943257 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:14.943219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:16.648105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:16.648073 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:20.950260 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:20.950228 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:21.750745 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:21.750715 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:22.225458 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:22.225425 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:26.749520 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:26.749483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:30.827158 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:30.827118 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:31.863295 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:31.863265 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:35.560731 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:55:35.560696 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:55:36.947309 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:36.947273 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:39.287712 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:39.287674 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:42.027168 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:42.027106 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:43.666869 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.666838 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 08:55:43.670809 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.670793 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.725313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.725286 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\"" Nov 03 08:55:43.743833 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.743806 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.743942 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.743842 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.743942 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.743877 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.743974 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744018 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.744008 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744117 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.744099 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.744187 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744242 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.744213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.744274 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.744248 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65h4m\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.777529 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.777506 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 08:55:43.844905 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.844882 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.844919 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.844942 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.844996 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845076 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845133 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65h4m\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845200 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845240 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845517 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845381 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845517 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845615 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845576 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845670 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.845896 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.845844 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.847488 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.847468 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.847592 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.847581 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.903931 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.903905 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.904051 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.904022 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65h4m\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:43.984386 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:43.984320 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:44.410427 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:44.410401 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 08:55:44.412180 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:55:44.412126 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f87c58b_6851_4213_9570_ea352b8853a4.slice/crio-783cc8bdd7d3878747767fc680502b7e4b6e5fa5af5519775c8a3f5de7e4c8d1 WatchSource:0}: Error finding container 783cc8bdd7d3878747767fc680502b7e4b6e5fa5af5519775c8a3f5de7e4c8d1: Status 404 returned error can't find the container with id 783cc8bdd7d3878747767fc680502b7e4b6e5fa5af5519775c8a3f5de7e4c8d1 Nov 03 08:55:44.414432 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:44.414403 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:55:44.414508 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:44.414458 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:55:44.414508 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:44.414486 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:55:45.014452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:45.014419 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" event={"ID":"8f87c58b-6851-4213-9570-ea352b8853a4","Type":"ContainerStarted","Data":"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64"} Nov 03 08:55:45.014950 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:45.014515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" event={"ID":"8f87c58b-6851-4213-9570-ea352b8853a4","Type":"ContainerStarted","Data":"783cc8bdd7d3878747767fc680502b7e4b6e5fa5af5519775c8a3f5de7e4c8d1"} Nov 03 08:55:45.139092 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:45.139042 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podStartSLOduration=2.139029368 podStartE2EDuration="2.139029368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:55:45.136929046 +0000 UTC m=+968.804969779" watchObservedRunningTime="2026-11-03 08:55:45.139029368 +0000 UTC m=+968.807070106" Nov 03 08:55:45.995164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:45.987433 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:46.006165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:46.002264 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:55:46.006165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:46.002323 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 08:55:46.993161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:46.987431 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:55:46.993161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:46.987497 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 08:55:47.061123 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:47.061093 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:47.999166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:47.992467 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:55:47.999166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:47.992524 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 08:55:48.988164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:48.987454 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:55:48.988164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:48.987509 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 08:55:49.990766 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:49.990729 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 08:55:49.991415 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:49.991384 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 08:55:51.515684 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:51.515649 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:51.516109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:51.516066 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:51.516895 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:51.516879 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 08:55:52.143557 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:52.143523 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:55:57.238011 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:55:57.237965 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:02.267403 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:02.267373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:03.005227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:03.005197 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:05.560630 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:56:05.560592 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:56:07.271418 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:07.271387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:12.349378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:12.349348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:14.091349 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.091316 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 08:56:14.095215 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.095196 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.167477 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.167447 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:14.185493 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185467 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.185659 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185508 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.185659 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185635 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.185769 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185680 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.185824 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185761 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.185879 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.185861 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8ww\" (UniqueName: \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.199283 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.199261 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"premium-w0-910d50-kserve-self-signed-certs\"" Nov 03 08:56:14.201821 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.201801 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 08:56:14.286722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286688 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.286722 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286725 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286753 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8ww\" (UniqueName: \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286889 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.286930 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287250 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.287115 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287343 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.287320 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.287377 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.287360 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.289161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.289124 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.289428 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.289412 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.349630 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.349570 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8ww\" (UniqueName: \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\") pod \"premium-w0-910d50-kserve-76cdbd64d7-n6v5b\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.407173 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.407125 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:14.804397 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:14.804374 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 08:56:14.805828 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:56:14.805803 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde7e967f_e4a7_4a65_bf67_53245d5cce74.slice/crio-950fb9e0999f92defd2d1596a6f2a0f1ba306093a0502caaa88bf8be6b8b3ecb WatchSource:0}: Error finding container 950fb9e0999f92defd2d1596a6f2a0f1ba306093a0502caaa88bf8be6b8b3ecb: Status 404 returned error can't find the container with id 950fb9e0999f92defd2d1596a6f2a0f1ba306093a0502caaa88bf8be6b8b3ecb Nov 03 08:56:15.183639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:15.183602 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" event={"ID":"de7e967f-e4a7-4a65-bf67-53245d5cce74","Type":"ContainerStarted","Data":"799ba95f65474fc1fcd9ef3e7e0b47bfdc45f22e0afaa17545d1d392cc231dfe"} Nov 03 08:56:15.183639 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:15.183645 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" event={"ID":"de7e967f-e4a7-4a65-bf67-53245d5cce74","Type":"ContainerStarted","Data":"950fb9e0999f92defd2d1596a6f2a0f1ba306093a0502caaa88bf8be6b8b3ecb"} Nov 03 08:56:15.303009 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:15.302958 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podStartSLOduration=1.302943637 podStartE2EDuration="1.302943637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:15.30146154 +0000 UTC m=+998.969502271" watchObservedRunningTime="2026-11-03 08:56:15.302943637 +0000 UTC m=+998.970984403" Nov 03 08:56:15.407711 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:15.407685 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:17.383450 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:17.383422 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:18.184836 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:18.184795 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:56:18.185157 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:18.185094 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" containerID="cri-o://cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857" gracePeriod=30 Nov 03 08:56:22.474845 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:22.474807 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:22.611300 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:22.611236 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:23.444186 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.444165 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:56:23.484237 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484130 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484253 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484279 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484317 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484354 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484394 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484441 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484525 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484563 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\") pod \"4d103670-9db2-4e56-8b59-c58812a52487\" (UID: \"4d103670-9db2-4e56-8b59-c58812a52487\") " Nov 03 08:56:23.484676 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.485212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484693 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.485212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484946 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.485212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.484976 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 08:56:23.485318 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.485265 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.485318 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.485288 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.485318 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.485304 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.485420 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.485318 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.485420 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.485331 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d103670-9db2-4e56-8b59-c58812a52487-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.486617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.486592 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 08:56:23.486723 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.486639 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:56:23.486974 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.486951 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "kube-api-access-v2thb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 08:56:23.487225 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.487210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo" pod "4d103670-9db2-4e56-8b59-c58812a52487" (UID: "4d103670-9db2-4e56-8b59-c58812a52487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 08:56:23.585954 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.585920 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d103670-9db2-4e56-8b59-c58812a52487-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.585954 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.585956 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d103670-9db2-4e56-8b59-c58812a52487-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.586111 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.585970 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2thb\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-kube-api-access-v2thb\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:23.586111 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:23.585980 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d103670-9db2-4e56-8b59-c58812a52487-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 08:56:24.229717 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.229681 4543 generic.go:356] "Generic (PLEG): container finished" podID="4d103670-9db2-4e56-8b59-c58812a52487" containerID="cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857" exitCode=0 Nov 03 08:56:24.229717 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.229722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" event={"ID":"4d103670-9db2-4e56-8b59-c58812a52487","Type":"ContainerDied","Data":"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857"} Nov 03 08:56:24.229951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.229741 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" Nov 03 08:56:24.229951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.229788 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm" event={"ID":"4d103670-9db2-4e56-8b59-c58812a52487","Type":"ContainerDied","Data":"5c9badd8f2609075f96df881b3d0b366731e6e3f74ab2fab768db2cd0eb27159"} Nov 03 08:56:24.229951 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.229809 4543 scope.go:122] "RemoveContainer" containerID="cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857" Nov 03 08:56:24.239648 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.239629 4543 scope.go:122] "RemoveContainer" containerID="cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857" Nov 03 08:56:24.239916 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:56:24.239898 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857\": container with ID starting with cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857 not found: ID does not exist" containerID="cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857" Nov 03 08:56:24.239962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.239925 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857"} err="failed to get container status \"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857\": rpc error: code = NotFound desc = could not find container \"cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857\": container with ID starting with cddd02cff74971acb20814f4b027588cebd7609c31fb8a65c383c1b2b2ffe857 not found: ID does not exist" Nov 03 08:56:24.408109 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.408083 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:24.420227 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.420200 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:24.452183 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.452155 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:56:24.502625 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:24.502564 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h65zm"] Nov 03 08:56:25.247073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:25.247047 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 08:56:26.452212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:26.452182 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d103670-9db2-4e56-8b59-c58812a52487" path="/var/lib/kubelet/pods/4d103670-9db2-4e56-8b59-c58812a52487/volumes" Nov 03 08:56:27.549832 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:27.549797 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:31.171981 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:31.171940 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:32.544330 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:32.544297 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:35.560730 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:56:35.560688 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:56:37.531991 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:37.531953 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:37.688343 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:37.688302 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:37.817170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:37.817055 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:42.531167 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:42.531101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:42.682619 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:42.682587 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:42.811883 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:42.811806 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:42.942805 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:42.942773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:47.113910 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:47.113831 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:47.575040 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:47.574955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:52.929363 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:52.929319 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:56:58.034209 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:56:58.034177 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:03.044044 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:03.044008 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:05.560759 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:57:05.560721 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:57:08.147147 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:08.147102 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:13.139350 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:13.139291 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:14.345210 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:14.345180 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:18.139687 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.139648 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:18.668334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.668301 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 08:57:18.668776 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.668763 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" Nov 03 08:57:18.668826 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.668776 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" Nov 03 08:57:18.668875 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.668865 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d103670-9db2-4e56-8b59-c58812a52487" containerName="istio-proxy" Nov 03 08:57:18.672170 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.672129 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.718677 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.718652 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 08:57:18.722276 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.722260 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.741683 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741660 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741779 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741696 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741824 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741779 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741824 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741803 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741887 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741829 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741918 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741910 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741953 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741938 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741958 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.741985 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.741974 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58nt\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.774096 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.774072 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 08:57:18.835587 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.835560 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 08:57:18.843803 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.843771 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.843973 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.843913 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.843973 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.843948 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844070 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.843997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844155 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844117 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844195 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844225 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844272 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844303 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwf8\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844332 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844351 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844368 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844436 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844437 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844452 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844453 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844470 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58nt\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844487 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.844933 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844541 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.845078 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.844974 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.845156 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.845079 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.845242 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.845224 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.847627 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.847605 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.849103 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.849084 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.909838 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.909805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58nt\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.910756 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.910735 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-zs7bw\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945518 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwf8\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945555 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945584 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945628 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945672 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945732 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945797 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.945816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.946293 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947202 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.947178 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.947430 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.947661 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.947888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.947738 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.949758 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.949734 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.949956 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.949935 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:18.984928 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:18.984896 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:19.005624 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.005588 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:19.005888 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.005866 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwf8\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8572v\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:19.034095 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.034069 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:19.411740 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.411708 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 08:57:19.450238 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:57:19.450211 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb8d8a1_5124_42a2_a9ab_6d54df2d89ad.slice/crio-1e49b80430f664eecdb6117ba92706a256daaa873934e5b31eeb1fcbcd391a44 WatchSource:0}: Error finding container 1e49b80430f664eecdb6117ba92706a256daaa873934e5b31eeb1fcbcd391a44: Status 404 returned error can't find the container with id 1e49b80430f664eecdb6117ba92706a256daaa873934e5b31eeb1fcbcd391a44 Nov 03 08:57:19.452499 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.452471 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.452589 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.452534 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.452589 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.452562 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.468926 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.468906 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 08:57:19.470441 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:57:19.470418 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88989000_c596_4d8c_8eb7_4d0eb5bd86bc.slice/crio-d39f05aa823b44d06e14289e6a4bb614a5d80c38e5f8577e83f97c7a31037a75 WatchSource:0}: Error finding container d39f05aa823b44d06e14289e6a4bb614a5d80c38e5f8577e83f97c7a31037a75: Status 404 returned error can't find the container with id d39f05aa823b44d06e14289e6a4bb614a5d80c38e5f8577e83f97c7a31037a75 Nov 03 08:57:19.472980 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.472952 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.473073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.473023 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.473073 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.473065 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:57:19.562000 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.561955 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" event={"ID":"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad","Type":"ContainerStarted","Data":"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b"} Nov 03 08:57:19.562265 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.562070 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" event={"ID":"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad","Type":"ContainerStarted","Data":"1e49b80430f664eecdb6117ba92706a256daaa873934e5b31eeb1fcbcd391a44"} Nov 03 08:57:19.566616 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.566577 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" event={"ID":"88989000-c596-4d8c-8eb7-4d0eb5bd86bc","Type":"ContainerStarted","Data":"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0"} Nov 03 08:57:19.566781 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.566642 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" event={"ID":"88989000-c596-4d8c-8eb7-4d0eb5bd86bc","Type":"ContainerStarted","Data":"d39f05aa823b44d06e14289e6a4bb614a5d80c38e5f8577e83f97c7a31037a75"} Nov 03 08:57:19.701006 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.700887 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podStartSLOduration=1.7008692810000001 podStartE2EDuration="1.700869281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:19.698057557 +0000 UTC m=+1063.366098306" watchObservedRunningTime="2026-11-03 08:57:19.700869281 +0000 UTC m=+1063.368910019" Nov 03 08:57:19.990232 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:19.985969 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:20.042867 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:20.041976 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:20.045779 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:20.045733 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:20.045941 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:20.045793 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:21.001196 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.001112 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:21.001804 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.001771 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:21.036186 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.035381 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:21.036186 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.035462 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:21.611042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.610964 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:21.998221 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.992258 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:21.998221 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:21.992318 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:22.037284 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:22.037249 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:22.038805 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:22.038776 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:22.917042 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:22.917007 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:22.985501 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:22.985467 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:22.985801 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:22.985772 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:23.035487 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:23.035451 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:23.035771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:23.035742 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:23.335379 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:23.335295 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:23.990292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:23.990251 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:23.990587 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:23.990558 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:24.035036 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:24.034804 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:24.035036 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:24.034862 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:25.000161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:24.999307 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:25.000161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:24.999381 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:25.038263 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:25.038230 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:25.038527 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:25.038498 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:25.985604 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:25.985573 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:25.985786 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:25.985629 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:26.034930 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:26.034894 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:26.035336 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:26.034964 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:26.985747 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:26.985704 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 08:57:26.985911 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:26.985786 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 08:57:27.035465 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:27.035428 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 08:57:27.036057 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:27.036018 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 08:57:28.069797 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.068915 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:28.069797 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.069745 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:28.071741 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.071689 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 08:57:28.179229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.176697 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:28.179229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.177347 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:28.179229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.178330 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 08:57:28.241216 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.241128 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podStartSLOduration=10.241114374 podStartE2EDuration="10.241114374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:19.821541376 +0000 UTC m=+1063.489582115" watchObservedRunningTime="2026-11-03 08:57:28.241114374 +0000 UTC m=+1071.909155111" Nov 03 08:57:28.501318 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:28.501284 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:31.194292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:31.194256 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:57:31.194934 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:31.194883 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:57:31.561064 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:31.560988 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:33.739352 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:33.739315 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:35.560411 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:57:35.560374 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:57:38.779986 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:38.779955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:40.599240 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:40.599211 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:40.727055 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:40.727015 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:43.854889 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:43.854861 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:45.699161 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:45.697614 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:45.843166 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:45.840184 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:46.007219 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:45.998717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:49.277013 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:49.276978 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:54.436549 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:54.436517 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:57:59.131329 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:59.129246 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:57:59.131329 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:59.129332 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:57:59.234165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:59.230230 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 08:57:59.234165 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:59.230307 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 08:57:59.739089 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:57:59.739052 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:01.015976 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.015946 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 08:58:01.019680 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.019664 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051526 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051503 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051533 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051681 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051573 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051813 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051813 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051761 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051813 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051794 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051922 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051870 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4mv\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.051922 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.051904 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.074413 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.074389 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-77207e-openshift-default-dockercfg-rcrfh\"" Nov 03 08:58:01.127105 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.127076 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 08:58:01.153223 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153192 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153286 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153378 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153312 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153535 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153463 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153589 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153652 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153637 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153734 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153713 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4mv\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.153800 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.153786 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.154321 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.154247 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.154321 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.154298 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.154321 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.154302 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.154509 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.154398 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.154590 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.154570 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.156388 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.156344 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.156538 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.156523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.213060 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.213034 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.213212 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.213190 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4mv\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.332847 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.332766 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:01.759299 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:58:01.759269 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb467afe3_a4dd_48e0_b8bb_034efa1aa4cb.slice/crio-2f5dab0489caad6050fbad41026eb20d535695dcd675213ea90e6cbba4fd9c15 WatchSource:0}: Error finding container 2f5dab0489caad6050fbad41026eb20d535695dcd675213ea90e6cbba4fd9c15: Status 404 returned error can't find the container with id 2f5dab0489caad6050fbad41026eb20d535695dcd675213ea90e6cbba4fd9c15 Nov 03 08:58:01.761376 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.761345 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:58:01.761441 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.761409 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:58:01.761481 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.761437 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 08:58:01.770751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.770728 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 08:58:01.996723 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.996681 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" event={"ID":"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb","Type":"ContainerStarted","Data":"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b"} Nov 03 08:58:01.996913 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:01.996737 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" event={"ID":"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb","Type":"ContainerStarted","Data":"2f5dab0489caad6050fbad41026eb20d535695dcd675213ea90e6cbba4fd9c15"} Nov 03 08:58:02.119106 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:02.119055 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podStartSLOduration=2.119040958 podStartE2EDuration="2.119040958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:58:02.118454028 +0000 UTC m=+1105.786494767" watchObservedRunningTime="2026-11-03 08:58:02.119040958 +0000 UTC m=+1105.787081694" Nov 03 08:58:02.333290 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:02.333251 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:02.335028 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:02.335005 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:02.335189 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:02.335048 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:03.333554 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:03.333521 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:03.333934 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:03.333578 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:04.333119 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:04.333078 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:04.333334 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:04.333173 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:04.736431 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:04.736401 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:05.333998 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:05.333959 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:05.334229 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:05.334032 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:05.561346 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:58:05.561312 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:58:06.334118 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:06.334082 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:06.334573 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:06.334133 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:07.333804 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:07.333768 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Nov 03 08:58:07.333962 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:07.333833 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Nov 03 08:58:08.715208 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:08.714423 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:08.728009 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:08.715728 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:08.728009 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:08.720346 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 08:58:09.988164 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:09.984675 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:10.768464 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:10.768413 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:15.042829 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:15.042792 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:20.066682 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:20.066596 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:23.261325 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:23.261288 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:25.109436 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:25.109406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:30.135746 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:30.135703 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:31.233620 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.233583 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 08:58:31.237556 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.237534 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.309053 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.309021 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:31.341774 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.341737 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"openshift-service-ca.crt\"" Nov 03 08:58:31.341774 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.341748 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"kube-root-ca.crt\"" Nov 03 08:58:31.342005 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.341804 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"distinct-w2-77207e-kserve-self-signed-certs\"" Nov 03 08:58:31.342005 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.341820 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 08:58:31.342005 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.341902 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"default-dockercfg-czmfl\"" Nov 03 08:58:31.394243 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394188 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.394243 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394226 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq44j\" (UniqueName: \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.394502 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394328 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.394502 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394362 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.394502 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.394653 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.394528 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496204 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496085 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496204 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496195 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496260 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496283 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq44j\" (UniqueName: \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496430 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496370 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496653 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496538 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496653 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496573 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.496771 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.496741 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.498544 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.498523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.498765 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.498748 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.555644 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.555616 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq44j\" (UniqueName: \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\") pod \"distinct-w2-77207e-kserve-587c5db559-wmffp\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.852063 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.851987 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:31.907581 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:31.907543 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:32.225505 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:32.225478 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 08:58:32.227111 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:58:32.227086 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4681caf5_0ecd_4348_b7a9_eb70ac0e793c.slice/crio-87f87487aaea0c806ed6276e80d9cf3c93e2a325fddac4279180a2eb259d06ac WatchSource:0}: Error finding container 87f87487aaea0c806ed6276e80d9cf3c93e2a325fddac4279180a2eb259d06ac: Status 404 returned error can't find the container with id 87f87487aaea0c806ed6276e80d9cf3c93e2a325fddac4279180a2eb259d06ac Nov 03 08:58:32.347313 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:32.347283 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" event={"ID":"4681caf5-0ecd-4348-b7a9-eb70ac0e793c","Type":"ContainerStarted","Data":"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9"} Nov 03 08:58:32.347678 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:32.347329 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" event={"ID":"4681caf5-0ecd-4348-b7a9-eb70ac0e793c","Type":"ContainerStarted","Data":"87f87487aaea0c806ed6276e80d9cf3c93e2a325fddac4279180a2eb259d06ac"} Nov 03 08:58:32.464769 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:32.464720 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podStartSLOduration=1.464707291 podStartE2EDuration="1.464707291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:58:32.463232914 +0000 UTC m=+1136.131273651" watchObservedRunningTime="2026-11-03 08:58:32.464707291 +0000 UTC m=+1136.132748029" Nov 03 08:58:32.852797 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:32.852771 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:35.106300 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:35.106269 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:35.560678 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:58:35.560648 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:58:36.578619 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:36.578585 4543 scope.go:122] "RemoveContainer" containerID="23aeddb0cf409d9a644ba8248a1e766c9c50d35abf5d63bee6f1c1361f3d17c9" Nov 03 08:58:40.077984 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:40.077938 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:41.852575 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:41.852544 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:41.864958 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:41.864927 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:42.415204 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:42.415177 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 08:58:45.140715 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:45.140685 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:48.849444 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:48.849401 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:50.119770 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:50.119736 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:58:55.133623 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:58:55.133583 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:00.142240 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:00.142210 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:05.151669 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:05.151628 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:05.560371 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:59:05.560292 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:59:10.133381 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:10.133343 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:15.132809 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:15.132775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:20.131363 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:20.131327 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:21.029385 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.029350 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 08:59:21.033119 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.033104 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.091475 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.091444 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 08:59:21.116558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.116522 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:21.124435 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124408 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.124558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124474 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.124617 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124585 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.124788 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124769 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptm2\" (UniqueName: \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.124829 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124808 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.124867 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.124834 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.139112 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.139091 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"openshift-service-ca.crt\"" Nov 03 08:59:21.139112 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.139103 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"default-dockercfg-dftgt\"" Nov 03 08:59:21.139112 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.139085 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"kube-root-ca.crt\"" Nov 03 08:59:21.139637 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.139203 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"distinct-w3-98d5dd-kserve-self-signed-certs\"" Nov 03 08:59:21.226056 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226020 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptm2\" (UniqueName: \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226056 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226058 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226074 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226172 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226220 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226293 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226258 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226544 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226659 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.226719 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.226675 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.228649 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.228625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.228751 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.228714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.285878 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.285819 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptm2\" (UniqueName: \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\") pod \"distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.345622 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.345591 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:21.724455 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:21.724429 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 08:59:21.726220 ip-10-0-131-117 kubenswrapper[4543]: W1103 08:59:21.726192 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca58f7f4_98c5_4061_9ce3_95913f05e795.slice/crio-0e7900f0dc8ae8bac65ee15cc2eded4edde0f6a962c76e74b56ccd40284533ff WatchSource:0}: Error finding container 0e7900f0dc8ae8bac65ee15cc2eded4edde0f6a962c76e74b56ccd40284533ff: Status 404 returned error can't find the container with id 0e7900f0dc8ae8bac65ee15cc2eded4edde0f6a962c76e74b56ccd40284533ff Nov 03 08:59:22.616367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:22.616334 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" event={"ID":"ca58f7f4-98c5-4061-9ce3-95913f05e795","Type":"ContainerStarted","Data":"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3"} Nov 03 08:59:22.616367 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:22.616371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" event={"ID":"ca58f7f4-98c5-4061-9ce3-95913f05e795","Type":"ContainerStarted","Data":"0e7900f0dc8ae8bac65ee15cc2eded4edde0f6a962c76e74b56ccd40284533ff"} Nov 03 08:59:22.735473 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:22.735425 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podStartSLOduration=2.735410429 podStartE2EDuration="2.735410429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:59:22.732961958 +0000 UTC m=+1186.401002700" watchObservedRunningTime="2026-11-03 08:59:22.735410429 +0000 UTC m=+1186.403451204" Nov 03 08:59:23.346561 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:23.346532 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:23.600514 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:23.600441 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:25.332292 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:25.332263 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:26.910126 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:26.910085 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:30.349558 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:30.349526 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:31.346176 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:31.346120 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:31.358414 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:31.358390 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:31.676779 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:31.676757 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 08:59:32.245736 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:32.245701 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:35.358503 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:35.358465 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:35.566370 ip-10-0-131-117 kubenswrapper[4543]: E1103 08:59:35.566333 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 08:59:36.563981 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:36.563956 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:59:36.566627 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:36.566606 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 08:59:40.350929 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:40.350896 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:42.474733 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:42.474703 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:45.507449 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:45.507390 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:50.427818 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:50.427743 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:52.191806 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:52.191769 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 08:59:55.540374 ip-10-0-131-117 kubenswrapper[4543]: I1103 08:59:55.540346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:00.627233 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:00.627193 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:05.561234 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:00:05.561193 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:00:05.640620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:05.640588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:10.636440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:10.636395 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:15.641758 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:15.641722 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:20.744923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:20.744884 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:21.115504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:21.115473 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:23.945097 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:23.945061 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:25.831989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:25.831955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:27.248618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:27.248576 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:30.881320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:30.881281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:32.191672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.191638 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:00:32.195829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.195809 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.283772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.283745 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.283929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.283860 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.283929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.283916 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.283964 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.284019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.284049 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.284191 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.284241 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.284322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.284294 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.299674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.299643 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:00:32.385219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385251 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385287 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385311 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385355 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385459 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385505 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385680 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385810 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.385945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385897 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.386019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.385963 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.387825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.387804 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.388021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.388004 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.444894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.444818 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.445165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.445124 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.508745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.508716 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:32.600373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.600343 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:32.943111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.943077 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:00:32.943825 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:00:32.943802 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939cd6ec_76fd_400b_a6e9_5f57fe037a7f.slice/crio-9e8faee225575b9efc31e24f3ca2e33ebf1a8893ab312c4a269595bdd2039224 WatchSource:0}: Error finding container 9e8faee225575b9efc31e24f3ca2e33ebf1a8893ab312c4a269595bdd2039224: Status 404 returned error can't find the container with id 9e8faee225575b9efc31e24f3ca2e33ebf1a8893ab312c4a269595bdd2039224 Nov 03 09:00:32.946083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.946054 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:00:32.946170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.946117 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:00:32.946170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:32.946161 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:00:33.010498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:33.010464 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" event={"ID":"939cd6ec-76fd-400b-a6e9-5f57fe037a7f","Type":"ContainerStarted","Data":"9e8faee225575b9efc31e24f3ca2e33ebf1a8893ab312c4a269595bdd2039224"} Nov 03 09:00:34.017240 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:34.017200 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" event={"ID":"939cd6ec-76fd-400b-a6e9-5f57fe037a7f","Type":"ContainerStarted","Data":"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12"} Nov 03 09:00:34.237067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:34.237015 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podStartSLOduration=2.236999172 podStartE2EDuration="2.236999172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:34.186356645 +0000 UTC m=+1257.854397385" watchObservedRunningTime="2026-11-03 09:00:34.236999172 +0000 UTC m=+1257.905039911" Nov 03 09:00:34.509987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:34.509957 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:34.511478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:34.511452 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:34.511598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:34.511510 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:35.510605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:35.510566 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:35.511239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:35.511204 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:35.581121 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:00:35.581086 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:00:35.943622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:35.943587 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:36.511212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:36.509993 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:36.511212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:36.510052 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:37.509365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:37.509334 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:37.509564 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:37.509385 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:38.510078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:38.510048 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:38.510520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:38.510099 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:39.509488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:39.509446 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 03 09:00:39.509692 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:39.509541 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 03 09:00:40.513992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:40.513960 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:40.514410 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:40.514259 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:40.515099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:40.515084 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:00:40.974006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:40.973977 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:43.233534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:43.233503 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:43.362253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:43.362223 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:45.953396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:45.953354 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:50.955946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:50.955910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:00:56.134155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:00:56.134110 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:01.224924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:01.224888 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:05.560540 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:01:05.560500 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:01:06.144231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:06.144200 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:09.770314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:09.770279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:11.133319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.133284 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:11.836280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.836248 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:01:11.840355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.840336 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.863969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.863943 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.863980 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864105 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864132 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864211 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864250 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdnx\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864294 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864370 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.864420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.864396 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.941726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.941694 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:01:11.965330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965342 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965402 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965438 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965529 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965574 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965641 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.965691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965680 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdnx\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965758 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965778 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.965832 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.966027 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.966033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.966321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.966190 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.968122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.968091 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:11.968316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:11.968297 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:12.024828 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.024800 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:12.024996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.024911 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdnx\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:12.153252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.153228 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:12.342771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.342737 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:12.580204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.580174 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:01:12.581413 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:01:12.581391 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb60d98_08a4_48da_9642_0d3abed943c8.slice/crio-32eece940b1043f2908ce7a106d9b10db64a5c1dd88ccd0aef9ef76825166fe3 WatchSource:0}: Error finding container 32eece940b1043f2908ce7a106d9b10db64a5c1dd88ccd0aef9ef76825166fe3: Status 404 returned error can't find the container with id 32eece940b1043f2908ce7a106d9b10db64a5c1dd88ccd0aef9ef76825166fe3 Nov 03 09:01:12.583668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.583637 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:12.583744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.583702 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:12.583744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:12.583728 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:13.308237 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:13.308181 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" event={"ID":"deb60d98-08a4-48da-9642-0d3abed943c8","Type":"ContainerStarted","Data":"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e"} Nov 03 09:01:13.308752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:13.308283 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" event={"ID":"deb60d98-08a4-48da-9642-0d3abed943c8","Type":"ContainerStarted","Data":"32eece940b1043f2908ce7a106d9b10db64a5c1dd88ccd0aef9ef76825166fe3"} Nov 03 09:01:13.431072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:13.431009 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podStartSLOduration=2.430995415 podStartE2EDuration="2.430995415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:13.427937775 +0000 UTC m=+1297.095978515" watchObservedRunningTime="2026-11-03 09:01:13.430995415 +0000 UTC m=+1297.099036154" Nov 03 09:01:13.671023 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:13.670985 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:14.154244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:14.154212 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:14.155802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:14.155775 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:14.155935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:14.155825 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:15.154133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:15.154095 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:15.154647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:15.154212 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:16.153724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:16.153692 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:16.153880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:16.153746 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:16.242874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:16.242839 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:17.154380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:17.154297 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:17.154380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:17.154349 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:18.154515 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:18.154484 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:18.154927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:18.154547 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:19.154072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:19.154040 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 09:01:19.154268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:19.154102 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 09:01:20.158697 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:20.158669 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:20.159175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:20.159124 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:20.159957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:20.159937 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:01:21.263434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:21.263403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:24.529875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:24.529837 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:26.560300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:26.560262 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:27.593267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:27.593230 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:29.967278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:29.967248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:30.101757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:30.101723 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:30.233672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:30.233591 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:31.679499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:31.679466 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:32.634638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.634604 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:01:32.638621 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.638605 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.684954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.684926 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:01:32.688756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.688737 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.767458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767422 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.767608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767553 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.767608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767596 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.767696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767628 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.767696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767653 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.767822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rpd\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.767889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767875 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.767930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.767904 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.768159 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768094 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768159 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768120 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768246 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768294 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.768429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768413 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768448 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.768507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768477 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.768578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768526 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2dw\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.768677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.768658 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.783670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.783643 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:01:32.839025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.838988 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:01:32.870342 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870309 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870342 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870344 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870383 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870406 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870431 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870456 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870488 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rpd\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870542 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870564 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870645 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870691 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870715 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870761 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870794 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870861 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870883 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870893 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870856 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.870917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870921 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.871553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.870947 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2dw\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.871553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.871100 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.871553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.871260 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.871553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.871476 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.871553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.871500 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.872460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.871982 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.872460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.872029 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.872460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.872055 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.874763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.874731 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.874763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.874755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.874955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.874930 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.876217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.876192 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.930420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.930361 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2dw\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.930533 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.930495 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rpd\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.931604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.931579 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:32.931700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.931665 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:32.938543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.938520 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:32.952098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:32.952080 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:33.001607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.001565 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:33.385817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.385793 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:01:33.387282 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:01:33.387256 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a7ea5e_d888_4edb_9c00_1cd3ee6f7511.slice/crio-1a04584829ee22b60cc09636987b5ea7c34c570eaf389eebe03be6b9c7ab0670 WatchSource:0}: Error finding container 1a04584829ee22b60cc09636987b5ea7c34c570eaf389eebe03be6b9c7ab0670: Status 404 returned error can't find the container with id 1a04584829ee22b60cc09636987b5ea7c34c570eaf389eebe03be6b9c7ab0670 Nov 03 09:01:33.389695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.389666 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.389774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.389726 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.389774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.389757 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.439634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.439611 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:01:33.441239 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:01:33.441216 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea552a0_00d9_400d_8dd2_958d1df54ca5.slice/crio-8271d2db6f07a35fb43cb262771033caace2e8f8b85a85b8c39c232bd75e0705 WatchSource:0}: Error finding container 8271d2db6f07a35fb43cb262771033caace2e8f8b85a85b8c39c232bd75e0705: Status 404 returned error can't find the container with id 8271d2db6f07a35fb43cb262771033caace2e8f8b85a85b8c39c232bd75e0705 Nov 03 09:01:33.443981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.443944 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.444085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.444027 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.444085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.444071 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:33.463187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.463134 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" event={"ID":"8ea552a0-00d9-400d-8dd2-958d1df54ca5","Type":"ContainerStarted","Data":"8271d2db6f07a35fb43cb262771033caace2e8f8b85a85b8c39c232bd75e0705"} Nov 03 09:01:33.465489 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:33.465462 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" event={"ID":"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511","Type":"ContainerStarted","Data":"1a04584829ee22b60cc09636987b5ea7c34c570eaf389eebe03be6b9c7ab0670"} Nov 03 09:01:34.313178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.313055 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:34.473126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.473091 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" event={"ID":"8ea552a0-00d9-400d-8dd2-958d1df54ca5","Type":"ContainerStarted","Data":"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87"} Nov 03 09:01:34.474908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.474884 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" event={"ID":"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511","Type":"ContainerStarted","Data":"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f"} Nov 03 09:01:34.839806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.839744 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podStartSLOduration=2.839728124 podStartE2EDuration="2.839728124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:34.838410682 +0000 UTC m=+1318.506451454" watchObservedRunningTime="2026-11-03 09:01:34.839728124 +0000 UTC m=+1318.507768927" Nov 03 09:01:34.840350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.840313 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podStartSLOduration=2.8403038609999998 podStartE2EDuration="2.840303861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:34.719407634 +0000 UTC m=+1318.387448397" watchObservedRunningTime="2026-11-03 09:01:34.840303861 +0000 UTC m=+1318.508344605" Nov 03 09:01:34.952756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.952725 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:34.954088 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.954065 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:34.954154 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:34.954110 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:35.016169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:35.012759 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:35.017694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:35.017253 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:35.017694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:35.017319 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:35.562080 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:01:35.562038 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:01:35.953695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:35.953656 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:35.954057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:35.954026 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:36.009168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:36.005742 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:36.009168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:36.005831 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:36.953234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:36.953199 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:36.953635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:36.953243 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:37.003164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:36.993887 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:37.003450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:37.003188 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:37.003450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:37.003254 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:37.952613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:37.952570 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:37.952792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:37.952625 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:38.006164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:38.005521 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:38.006164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:38.005600 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:38.953289 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:38.953260 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:38.953545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:38.953520 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:39.009487 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:39.009454 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:39.010071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:39.010041 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:39.953103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:39.953065 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 09:01:39.953306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:39.953132 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 09:01:40.006715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:40.006674 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 09:01:40.007015 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:40.006961 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 09:01:40.958079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:40.958050 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:40.958554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:40.958527 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:40.959129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:40.959114 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:01:41.007331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:41.007280 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:41.008051 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:41.008033 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:41.008713 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:41.008693 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:01:42.114835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.114804 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:01:42.119880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.119864 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.147661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.147636 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:42.211703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.211652 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:01:42.266198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266163 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266286 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266361 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266419 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266485 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266593 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266693 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzwr7\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266732 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.266838 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.266764 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.367729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367656 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.367729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367723 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.367947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367771 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.367947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367838 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzwr7\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.367947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367874 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.367997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368112 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368235 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368196 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368269 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368319 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368410 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368554 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368669 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368549 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.368758 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.368736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.370375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.370353 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.370542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.370524 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.426796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.426766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzwr7\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.426796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.426784 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.433666 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.433644 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:42.859994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.859959 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:01:42.870019 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:01:42.869985 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785907e1_e6ca_4037_ab6e_d2e9ab4ee320.slice/crio-452a04b6a946c13eac80e75c7c594df21926987d4b1822b5c8e4118b7f49e578 WatchSource:0}: Error finding container 452a04b6a946c13eac80e75c7c594df21926987d4b1822b5c8e4118b7f49e578: Status 404 returned error can't find the container with id 452a04b6a946c13eac80e75c7c594df21926987d4b1822b5c8e4118b7f49e578 Nov 03 09:01:42.872025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.871987 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:42.872115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.872057 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:42.872115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:42.872084 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:43.603166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:43.600267 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" event={"ID":"785907e1-e6ca-4037-ab6e-d2e9ab4ee320","Type":"ContainerStarted","Data":"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6"} Nov 03 09:01:43.603166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:43.600339 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" event={"ID":"785907e1-e6ca-4037-ab6e-d2e9ab4ee320","Type":"ContainerStarted","Data":"452a04b6a946c13eac80e75c7c594df21926987d4b1822b5c8e4118b7f49e578"} Nov 03 09:01:43.757911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:43.757878 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:43.906590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:43.906550 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:44.165119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:44.165034 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:44.434212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:44.434115 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:44.435892 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:44.435864 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:44.436008 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:44.435911 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:45.439167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:45.437976 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:45.439167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:45.438065 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:46.441169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:46.437521 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:46.441169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:46.437582 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:47.515166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.495499 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:47.531512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.531468 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:01:47.550076 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.549288 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:47.550391 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.550359 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:47.685389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.685323 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 09:01:47.823363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:47.823252 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.588370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.588312 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.624167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.590680 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.635310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.609934 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:48.635310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.635246 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:48.724166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.690537 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:48.803550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.784647 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:01:48.803550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.784724 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:01:48.803550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.784761 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:01:48.847043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.846945 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 09:01:48.864739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.864710 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:01:48.984611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:48.984541 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podStartSLOduration=6.98452013 podStartE2EDuration="6.98452013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:43.848816121 +0000 UTC m=+1327.516856860" watchObservedRunningTime="2026-11-03 09:01:48.98452013 +0000 UTC m=+1332.652560870" Nov 03 09:01:49.342194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.315559 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:49.546401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.511444 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:01:49.546401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.514679 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:49.546401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.514728 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:49.618273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.618229 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.632262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.632220 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:49.862168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:49.860771 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 09:01:50.539166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.538273 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:50.582404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.582338 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:50.582696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.582665 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:50.677537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.676427 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:01:50.677537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.676824 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 09:01:50.864268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:50.864219 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:51.123829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.123736 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:01:51.187110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.187080 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:01:51.187480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.187461 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.273173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.270634 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:01:51.353557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.353529 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366690 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366748 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366798 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366880 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366928 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp444\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.366961 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.367085 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.367131 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.372693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.372302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.374719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.374568 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:01:51.437161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.436538 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:51.437161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.436606 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:51.473649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473609 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.473956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473675 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.473956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473707 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.473956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473850 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.473956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473898 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.473956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.473947 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474061 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474109 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp444\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474163 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474425 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474531 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.474760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.474671 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.476275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.476246 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.476782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.476758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.538510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.538473 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.538661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.538520 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp444\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:51.677836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:51.677746 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:52.149545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.149521 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:01:52.150097 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:01:52.150074 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2572ad_7c80_42e2_9ea2_7a6e695da0e5.slice/crio-0897455c9a62661a8a6eeb9cf746497efcaa43e842a87ea0003271abd2790dad WatchSource:0}: Error finding container 0897455c9a62661a8a6eeb9cf746497efcaa43e842a87ea0003271abd2790dad: Status 404 returned error can't find the container with id 0897455c9a62661a8a6eeb9cf746497efcaa43e842a87ea0003271abd2790dad Nov 03 09:01:52.152595 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.152557 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:52.152678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.152627 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:52.152678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.152657 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:01:52.177087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.177062 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" event={"ID":"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5","Type":"ContainerStarted","Data":"0897455c9a62661a8a6eeb9cf746497efcaa43e842a87ea0003271abd2790dad"} Nov 03 09:01:52.434987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.434900 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:52.434987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:52.434951 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:53.148534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.148471 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:53.191165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.190886 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" event={"ID":"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5","Type":"ContainerStarted","Data":"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef"} Nov 03 09:01:53.363857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.363804 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podStartSLOduration=3.363789784 podStartE2EDuration="3.363789784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:53.360394895 +0000 UTC m=+1337.028435636" watchObservedRunningTime="2026-11-03 09:01:53.363789784 +0000 UTC m=+1337.031830523" Nov 03 09:01:53.434586 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.434504 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 09:01:53.434586 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.434552 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 09:01:53.677900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.677855 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:53.679802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.679768 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:53.679956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.679826 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:53.820120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:53.819997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:54.551840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:54.551777 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:54.678115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:54.678085 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:54.678321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:54.678159 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:55.438368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.438330 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:01:55.438540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.438386 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:01:55.439436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.439411 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:55.439753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.439735 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:55.440439 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.440421 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:01:55.678754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.678723 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:55.679116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:55.678768 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:56.678243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:56.678202 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:56.678461 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:56.678264 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:57.678218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:57.678183 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:57.678675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:57.678253 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:58.678082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:58.678041 4543 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 09:01:58.678431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:58.678097 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 09:01:59.682468 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:59.682435 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:59.682881 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:59.682804 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:59.683369 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:59.683349 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:01:59.755226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:59.755199 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:01:59.884895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:01:59.884864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:02.713262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:02.713223 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:04.753293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:04.753258 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:05.560626 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:02:05.560590 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:02:07.490329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:07.490266 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:07.682392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:07.682316 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:07.949277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:07.949234 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.391166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:08.380796 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.502174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:08.499421 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:08.659294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:08.643076 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:02:08.692952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:08.687103 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 09:02:08.692952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:08.687829 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:02:09.031233 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.028363 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 09:02:09.031233 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.029153 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:09.390165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.383221 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.390165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.383884 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:09.643223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.578469 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.705253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.695275 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:09.812185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.799673 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:02:09.812185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:09.799781 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:10.068335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.068222 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:10.394685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.393890 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:02:10.589256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.589215 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 09:02:10.702304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.702178 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:10.898030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.897991 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:02:10.992447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:10.992360 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:02:11.022974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:11.019360 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:11.051164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:11.041071 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:11.152304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:11.152170 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:11.225715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:11.225690 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:16.178587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:16.178557 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:17.090280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.090239 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:02:17.095721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.095695 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128437 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsm95\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128578 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128618 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128636 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128659 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128724 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128759 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128896 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.128947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.128914 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.205298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.205261 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:02:17.229593 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229556 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.229764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229612 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.229764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229634 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.229764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229656 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.229764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229676 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.229764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229700 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229847 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.229937 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsm95\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.230006 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.230159 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.230257 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.230547 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.230525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.231153 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.231111 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.232370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.232341 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.232660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.232643 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.290918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.290890 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.291099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.291006 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsm95\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.409979 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.409943 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:17.849113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.849083 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:02:17.849113 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:02:17.849101 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb417ea79_970a_41cc_93df_d6547cf8c414.slice/crio-9bc5176a96caa6bc31bd0a88df94dfc7deff65d6697b6aaec7b5ff5cc001f196 WatchSource:0}: Error finding container 9bc5176a96caa6bc31bd0a88df94dfc7deff65d6697b6aaec7b5ff5cc001f196: Status 404 returned error can't find the container with id 9bc5176a96caa6bc31bd0a88df94dfc7deff65d6697b6aaec7b5ff5cc001f196 Nov 03 09:02:17.851541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.851509 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:02:17.851630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.851573 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:02:17.851630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.851601 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:02:17.935703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.935668 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" event={"ID":"b417ea79-970a-41cc-93df-d6547cf8c414","Type":"ContainerStarted","Data":"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158"} Nov 03 09:02:17.935827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:17.935738 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" event={"ID":"b417ea79-970a-41cc-93df-d6547cf8c414","Type":"ContainerStarted","Data":"9bc5176a96caa6bc31bd0a88df94dfc7deff65d6697b6aaec7b5ff5cc001f196"} Nov 03 09:02:18.057270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:18.057219 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podStartSLOduration=1.0572067569999999 podStartE2EDuration="1.057206757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:18.05601961 +0000 UTC m=+1361.724060349" watchObservedRunningTime="2026-11-03 09:02:18.057206757 +0000 UTC m=+1361.725247496" Nov 03 09:02:18.411068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:18.411038 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:18.412439 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:18.412414 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:18.412578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:18.412467 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:19.411024 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:19.410979 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:19.411234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:19.411052 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:20.410673 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:20.410638 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:20.410850 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:20.410700 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:21.382707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:21.382672 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:21.410700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:21.410663 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:21.412444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:21.412382 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:22.412687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:22.412644 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:22.413371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:22.413317 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:22.915742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:22.915696 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.097408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.096231 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.401166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.400222 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.425165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.424527 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:23.425165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.424596 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:23.701328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.701225 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:23.918424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.918377 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 09:02:23.936195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:23.919427 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:02:24.199228 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.147600 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 09:02:24.199228 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.166268 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:24.408172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.401509 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:24.408172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.402311 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:24.433038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.433002 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:24.433662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.433629 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:24.737364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.671500 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:02:24.746524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.746349 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:24.756996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:24.756968 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:25.412020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:25.411979 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 09:02:25.412248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:25.412049 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 09:02:26.415046 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:26.415016 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:26.415520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:26.415453 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:26.416087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:26.416069 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:02:26.973429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:26.973397 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:27.942818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:27.942787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:28.543714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:28.543682 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:32.247366 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:32.247323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:33.288947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:33.288828 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:34.643230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:34.643198 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:35.562973 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:02:35.562935 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:02:36.387056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:36.387025 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:37.294213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:37.294049 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:37.865046 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:37.865002 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:38.555318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:38.555273 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:38.589306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:38.589265 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:38.696555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:38.695307 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 09:02:38.759512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:38.759462 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:38.760497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:38.760467 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:02:39.592308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.592258 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:39.593824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.593801 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:02:39.767292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.767237 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:39.767964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.767933 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 09:02:39.775162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.768797 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:39.775396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.775379 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:39.825278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.825239 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:02:39.933189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:39.928006 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:40.250288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:40.249372 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:02:40.271054 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:40.269260 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:02:40.274678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:40.274645 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:02:43.039880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:43.039835 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:43.933356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:43.933323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:44.103889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:44.103850 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:44.256334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:44.256247 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:47.247276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:47.247130 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:47.423684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:47.423650 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:48.599882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:48.599850 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:54.369242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:54.369205 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:54.504296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:54.504257 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:54.636320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:54.636248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:55.242747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:55.242712 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:55.732421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:55.732391 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:57.722166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:57.721823 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:57.968167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:57.966625 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:58.136166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:58.113827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:59.069167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.068813 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:59.219167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.218821 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:59.257288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.257240 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.274271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.274216 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:02:59.330386 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.330289 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:02:59.383727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.383685 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:59.531030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.530973 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:02:59.879451 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:02:59.879420 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:00.290336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:00.290241 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:00.316701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:00.316668 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:03:01.257166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:01.254238 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:01.365024 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:01.364978 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:03:01.394672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:01.365611 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 03 09:03:01.900165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:01.878267 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:02.123354 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.123317 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:02.258362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.258259 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:02.279277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.258984 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:03:02.415164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.414415 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:02.415164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.414780 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 09:03:02.450938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.441328 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:03:02.889833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:02.889788 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:03.083468 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:03.083441 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:03:03.155019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:03.154935 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:03:03.204735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:03.187301 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:03:03.230220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:03.224482 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:03.401659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:03.401618 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:05.561540 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:03:05.561504 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:03:05.853996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:05.853924 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:10.846114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:10.846075 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:11.721345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:11.721311 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:11.855197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:11.855167 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:15.828167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:15.813113 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:03:15.834152 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:15.834109 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:15.875606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:15.875573 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:03:15.927377 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:15.927343 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b0c653f4-serving-cert\"" Nov 03 09:03:16.117736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.106882 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.117736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.106955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4qdr\" (UniqueName: \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.117736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.107037 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.117736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.107076 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.172394 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.155950 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:16.204165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.202349 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:03:16.204165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.202416 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:03:16.244303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.217802 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.244303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.217879 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4qdr\" (UniqueName: \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.244303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.217961 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.244303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.218002 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.244303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.218936 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.296094 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:03:16.251529 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-serving-cert: secret "maas-api-e2e-mt-b0c653f4-serving-cert" not found Nov 03 09:03:16.296094 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:03:16.251612 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls podName:496d821e-6c79-473b-b7c7-fa54c575ae39 nodeName:}" failed. No retries permitted until 2026-11-03 09:03:16.751589811 +0000 UTC m=+1420.419630532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls") pod "maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" (UID: "496d821e-6c79-473b-b7c7-fa54c575ae39") : secret "maas-api-e2e-mt-b0c653f4-serving-cert" not found Nov 03 09:03:16.296094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.295259 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.417220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.383342 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4qdr\" (UniqueName: \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.417547 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.389259 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:16.752635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.752532 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:16.763908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.763871 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.766542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.766521 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:16.788435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:16.788407 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:17.067245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:17.067165 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:17.171723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:17.171695 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:03:17.172107 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:03:17.172086 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d821e_6c79_473b_b7c7_fa54c575ae39.slice/crio-0608c568aa44342e17d9353f0d64138e2b9e32b2afe0a33d8777ed31ef90171b WatchSource:0}: Error finding container 0608c568aa44342e17d9353f0d64138e2b9e32b2afe0a33d8777ed31ef90171b: Status 404 returned error can't find the container with id 0608c568aa44342e17d9353f0d64138e2b9e32b2afe0a33d8777ed31ef90171b Nov 03 09:03:17.173581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:17.173563 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:03:17.397087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:17.397042 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" event={"ID":"496d821e-6c79-473b-b7c7-fa54c575ae39","Type":"ContainerStarted","Data":"0608c568aa44342e17d9353f0d64138e2b9e32b2afe0a33d8777ed31ef90171b"} Nov 03 09:03:18.404434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:18.404350 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" event={"ID":"496d821e-6c79-473b-b7c7-fa54c575ae39","Type":"ContainerStarted","Data":"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1"} Nov 03 09:03:18.404434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:18.404418 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:21.441099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:21.441067 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:23.964413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:23.964381 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:24.416281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:24.416254 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:03:24.535418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:24.535369 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" podStartSLOduration=8.574061523 podStartE2EDuration="9.535356266s" totalImagesPullingTime="961.294743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:18.531282015 +0000 UTC m=+1422.199322754" watchObservedRunningTime="2026-11-03 09:03:24.535356266 +0000 UTC m=+1428.203397004" Nov 03 09:03:26.499239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:26.499209 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:28.268358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:28.268313 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:31.585166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:31.585047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:32.448167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.441241 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:03:32.472163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.467079 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.526194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.525440 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 09:03:32.526722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.526508 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 09:03:32.526722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.526572 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 09:03:32.526722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.526682 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9gqlj\"" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.582989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583047 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hk6\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583125 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583389 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583440 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.589177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.583476 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.636559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.636528 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:03:32.690284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.690245 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.702383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.702281 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.702651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.702611 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.702970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.702816 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.715458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.709116 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.715698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.715624 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hk6\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.715965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.715902 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.724418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.724348 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.793647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.770523 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.793647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.770751 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.953113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.953033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.953384 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.953276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hk6\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.973165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.968379 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:32.973165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:32.968817 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ldmbk\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:33.122711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:33.122680 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:33.753170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:33.750438 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:34.212955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:34.196392 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:03:34.564606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:34.564527 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" event={"ID":"53853238-48e6-4f0e-96ed-5b033097689e","Type":"ContainerStarted","Data":"ac4998f7e5553e9f8011560407af194d7685a6bd02e134b41dcb13a7e6130461"} Nov 03 09:03:35.064569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:35.064502 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:35.562446 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:03:35.562410 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:03:36.486400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:36.486366 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:36.641556 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:36.641524 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:36.771796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:36.771721 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:37.833169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:37.833121 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:03:37.833441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:37.833206 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:03:38.607663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:38.607631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" event={"ID":"53853238-48e6-4f0e-96ed-5b033097689e","Type":"ContainerStarted","Data":"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae"} Nov 03 09:03:38.607948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:38.607923 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:38.609799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:38.609778 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ldmbk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:03:38.609908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:38.609830 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:03:38.820064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:38.819999 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" podStartSLOduration=3.25138822 podStartE2EDuration="6.819946291s" totalImagesPullingTime="3.568558071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:38.779065725 +0000 UTC m=+1442.447106465" watchObservedRunningTime="2026-11-03 09:03:38.819946291 +0000 UTC m=+1442.487987033" Nov 03 09:03:39.617386 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:39.617357 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ldmbk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:03:39.617796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:39.617407 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:03:41.744343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:41.744307 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:42.618110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:42.618082 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:03:46.839689 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:46.839655 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:52.077285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:52.077249 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:03:57.236265 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:03:57.236236 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:00.945585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:00.945542 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:02.445249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:02.445220 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:05.342550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:05.342511 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:05.562069 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:04:05.562029 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:04:07.377021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:07.376992 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:07.859787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:07.859711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:08.009249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:08.001770 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:08.392894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:08.392855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:12.851348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:12.851311 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:13.034550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:13.034516 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:13.199131 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:13.199083 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:13.354354 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:13.354323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:16.126095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:16.126061 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:16.336397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:16.336358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:17.890798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:17.890722 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:04:17.899010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:17.898987 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.006310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.006271 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:04:18.035174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035107 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035191 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035330 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035515 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035380 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035515 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035458 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035521 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035639 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035697 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.035743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.035724 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdtg\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137152 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137106 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137186 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137209 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137240 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137284 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137320 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137348 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdtg\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137481 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137498 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137639 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137853 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.137936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.137917 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.138170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.138133 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.138256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.138240 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.139842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.139826 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.140086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.140060 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.196995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.196931 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.196995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.196940 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdtg\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.213701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.213679 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:18.548106 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.548008 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:18.658188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.658129 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:04:18.661731 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:04:18.661699 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0b6c34_6e16_4db3_a884_f9c3c3c8ee70.slice/crio-c7b37f20cef68dd9ad4b69d01ed027e0b974da06c7e91a7f579a331445dc8126 WatchSource:0}: Error finding container c7b37f20cef68dd9ad4b69d01ed027e0b974da06c7e91a7f579a331445dc8126: Status 404 returned error can't find the container with id c7b37f20cef68dd9ad4b69d01ed027e0b974da06c7e91a7f579a331445dc8126 Nov 03 09:04:18.664548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.664504 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:04:18.664714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.664584 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:04:18.664714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.664621 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:04:18.867553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.867513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" event={"ID":"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70","Type":"ContainerStarted","Data":"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7"} Nov 03 09:04:18.867724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.867583 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" event={"ID":"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70","Type":"ContainerStarted","Data":"c7b37f20cef68dd9ad4b69d01ed027e0b974da06c7e91a7f579a331445dc8126"} Nov 03 09:04:18.990981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:18.989466 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podStartSLOduration=1.9894467279999999 podStartE2EDuration="1.989446728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:18.989378591 +0000 UTC m=+1482.657419331" watchObservedRunningTime="2026-11-03 09:04:18.989446728 +0000 UTC m=+1482.657487468" Nov 03 09:04:19.214601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:19.214512 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:19.216303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:19.216276 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:19.216519 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:19.216494 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:20.214742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:20.214629 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:20.214742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:20.214692 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:21.215112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:21.215072 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:21.215703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:21.215153 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:22.214612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:22.214575 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:22.214785 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:22.214628 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:23.214679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:23.214636 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:23.215219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:23.214694 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:23.741394 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:23.741358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:24.214444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:24.214405 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:24.214655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:24.214463 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:25.214248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:25.214213 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 09:04:25.214619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:25.214264 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 09:04:25.419676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:25.419634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:26.119904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.119864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:26.218598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.218565 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:26.218982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.218845 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:26.219448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.219429 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:04:26.755687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.755654 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:26.970450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:26.970416 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:29.321767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:29.321736 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:30.822111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:30.822078 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:31.000513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:31.000468 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:31.144443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:31.144412 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:34.075987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:34.075944 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:34.231079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:34.231036 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:34.363789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:34.363758 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:34.945476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:34.945440 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:35.412080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:35.412044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:35.561869 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:04:35.561838 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:04:36.605770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:36.605290 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:04:36.608851 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:36.608824 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:04:38.245185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:38.245120 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:40.048575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:40.048538 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:45.118380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:45.118333 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:50.181438 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:50.181387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:55.104973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:55.104933 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:55.284786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:55.284731 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:55.415624 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:55.415588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:04:58.714192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:04:58.714150 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:00.458037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:00.458008 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:02.629017 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.628985 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:05:02.633067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.633051 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.733717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.733681 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:05:02.826318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826287 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f4c8\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826346 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826421 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826470 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826527 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826566 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826583 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826600 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.826854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.826818 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.927904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.927823 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.927904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.927858 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.927904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.927883 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928133 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f4c8\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928250 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928274 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928355 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928601 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928641 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928672 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928766 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.928927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.928775 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.930393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.930371 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.930855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.930831 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.987054 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.987026 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:02.987347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:02.987326 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f4c8\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:03.246559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:03.246480 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:03.675284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:03.675259 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:05:03.676027 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:05:03.675999 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27eff15c_f1fa_48cb_92c2_80ad44d06b40.slice/crio-13fb53755ae663b0b0a036f88339fe90d90ff3b307b73deffbbd4a1946d05d47 WatchSource:0}: Error finding container 13fb53755ae663b0b0a036f88339fe90d90ff3b307b73deffbbd4a1946d05d47: Status 404 returned error can't find the container with id 13fb53755ae663b0b0a036f88339fe90d90ff3b307b73deffbbd4a1946d05d47 Nov 03 09:05:03.681697 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:03.681650 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:03.681806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:03.681731 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:03.681806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:03.681774 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:04.224831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.224793 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" event={"ID":"27eff15c-f1fa-48cb-92c2-80ad44d06b40","Type":"ContainerStarted","Data":"bcff47c8e582f166c3d287281dfd2d1a7672751c06387f03521707d212c7a78c"} Nov 03 09:05:04.225028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.224855 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" event={"ID":"27eff15c-f1fa-48cb-92c2-80ad44d06b40","Type":"ContainerStarted","Data":"13fb53755ae663b0b0a036f88339fe90d90ff3b307b73deffbbd4a1946d05d47"} Nov 03 09:05:04.247358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.247328 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:04.248476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.248448 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:04.248619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.248494 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:04.346987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:04.346925 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podStartSLOduration=2.346905518 podStartE2EDuration="2.346905518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:05:04.343062273 +0000 UTC m=+1528.011103012" watchObservedRunningTime="2026-11-03 09:05:04.346905518 +0000 UTC m=+1528.014946254" Nov 03 09:05:05.247374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:05.247343 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:05.247782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:05.247393 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:05.497169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:05.497110 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:05.567824 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:05:05.567660 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:05:05.637909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:05.637870 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:06.246834 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:06.246800 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:06.247026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:06.246851 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:07.247525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:07.247493 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:07.247927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:07.247545 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:08.247464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:08.247429 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:08.247743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:08.247492 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:09.246868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:09.246831 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:09.247040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:09.246882 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:10.247787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:10.247753 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 09:05:10.248206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:10.247807 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 09:05:10.435803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:10.435770 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:11.250994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:11.250964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:11.251414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:11.251316 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:11.251922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:11.251907 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:05:15.803169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:15.789687 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:16.020999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:16.011215 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:16.150401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:16.150335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:16.215847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:16.198249 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:05:16.264198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:16.260228 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:05:16.264198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:16.260336 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:05:20.580882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:20.580837 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:21.103465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:21.103431 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:21.247560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:21.247512 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:23.639011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.638973 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:05:23.643403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.643381 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.698281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.698253 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-204dfc-openshift-default-dockercfg-rjz9h\"" Nov 03 09:05:23.735523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735493 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735522 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735675 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735774 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735803 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735837 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735928 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.735982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.735968 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znscm\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.753304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.753267 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:05:23.837055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837021 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znscm\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837241 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837189 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837241 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837218 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837266 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837326 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837438 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837493 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837530 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837601 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837689 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.837991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.837970 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.838132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.838110 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.839812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.839783 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.839999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.839979 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.896548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.896491 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.896652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.896619 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znscm\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:23.957779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:23.957755 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:24.386232 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:24.386206 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:05:24.387124 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:05:24.387092 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf483ea39_8bfa_424a_93c2_34e46689ff00.slice/crio-bef6c50f3467bbc81df0cd20a6df0e1f1156837f6fffec9abb15a3dc3a6525ea WatchSource:0}: Error finding container bef6c50f3467bbc81df0cd20a6df0e1f1156837f6fffec9abb15a3dc3a6525ea: Status 404 returned error can't find the container with id bef6c50f3467bbc81df0cd20a6df0e1f1156837f6fffec9abb15a3dc3a6525ea Nov 03 09:05:24.389623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:24.389591 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:24.389703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:24.389658 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:24.389703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:24.389687 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:05:24.430062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:24.430034 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" event={"ID":"f483ea39-8bfa-424a-93c2-34e46689ff00","Type":"ContainerStarted","Data":"bef6c50f3467bbc81df0cd20a6df0e1f1156837f6fffec9abb15a3dc3a6525ea"} Nov 03 09:05:25.438102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:25.438059 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" event={"ID":"f483ea39-8bfa-424a-93c2-34e46689ff00","Type":"ContainerStarted","Data":"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69"} Nov 03 09:05:25.958189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:25.958133 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:25.959452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:25.959417 4543 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 09:05:25.959612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:25.959478 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 09:05:26.129323 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:26.129287 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:26.958914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:26.958880 4543 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 09:05:26.959314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:26.958932 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 09:05:27.958587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:27.958559 4543 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 09:05:27.958749 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:27.958607 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 09:05:28.958353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:28.958319 4543 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 09:05:28.958718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:28.958375 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 09:05:29.958108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:29.958072 4543 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 09:05:29.958318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:29.958167 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 09:05:30.962740 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:30.962710 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:30.963168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:30.962995 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:30.963677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:30.963661 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:05:31.133065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:31.133017 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podStartSLOduration=8.133004279 podStartE2EDuration="8.133004279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:05:25.66496085 +0000 UTC m=+1549.333001594" watchObservedRunningTime="2026-11-03 09:05:31.133004279 +0000 UTC m=+1554.801045018" Nov 03 09:05:31.220355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:31.220265 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:34.422655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:34.422626 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:35.562578 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:05:35.562543 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:05:35.758457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:35.758427 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:36.304437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:36.304406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:41.339348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:41.339313 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:46.458457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:46.458426 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:51.626370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:51.626330 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:54.231695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:54.231660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:54.932299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:54.932247 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:55.601927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:55.600793 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:55.875923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:55.875889 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:05:56.870256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:05:56.870221 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:01.868220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:01.868190 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:05.561380 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:06:05.561345 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:06:06.925472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:06.925439 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:12.311655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:12.311616 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:12.651902 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:12.651871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:12.805283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:12.805245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:12.941625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:12.941536 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:17.558433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:17.558401 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:17.687773 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:17.687738 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:17.840671 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:17.840582 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:18.068786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:18.068750 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:20.659897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:20.659861 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:22.774016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:22.773981 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:26.535821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:26.535786 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:26.662124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:26.662082 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:27.698458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:27.698417 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:28.189271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:28.189241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:28.394507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:28.394465 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:29.499041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:29.499009 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:30.040931 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.040894 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:06:30.045184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.045131 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.077791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.077759 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078019 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078156 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078283 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078396 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g86p8\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078591 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078693 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.078828 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.078815 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.082373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.082343 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.114466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.112665 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-840559-openshift-default-dockercfg-7h7s5\"" Nov 03 09:06:30.153206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.149117 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:30.176993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.176960 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:06:30.188441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.187067 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.189036 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.188993 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.189239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.189222 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.189379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.189364 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.195257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.193480 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.195257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.192596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.195257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.192390 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.195598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.195578 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.195744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.195725 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g86p8\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.196005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.195977 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.196155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.196116 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.196752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.196718 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.201414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.196904 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.201634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.201607 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.206245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.205581 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.222195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.219736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.255545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.255011 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.282083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.270657 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g86p8\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\") pod \"e2e-api-a-840559-openshift-default-c66b5d46c-26b4z\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.361583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.361553 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:30.364480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.364435 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:30.802706 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.802679 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:06:30.803058 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:06:30.802934 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86e01d_e600_4cab_b543_6c39d5f1dac6.slice/crio-db0ba4b1c8c554d87dc5241a62c4444945061413512143d09005dd8275c3f98e WatchSource:0}: Error finding container db0ba4b1c8c554d87dc5241a62c4444945061413512143d09005dd8275c3f98e: Status 404 returned error can't find the container with id db0ba4b1c8c554d87dc5241a62c4444945061413512143d09005dd8275c3f98e Nov 03 09:06:30.805497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.805466 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:06:30.805566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.805530 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:06:30.805566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.805558 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:06:30.947050 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.947013 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerStarted","Data":"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2"} Nov 03 09:06:30.947277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:30.947076 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerStarted","Data":"db0ba4b1c8c554d87dc5241a62c4444945061413512143d09005dd8275c3f98e"} Nov 03 09:06:31.073208 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:31.073077 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podStartSLOduration=2.073058462 podStartE2EDuration="2.073058462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:31.068985078 +0000 UTC m=+1614.737025817" watchObservedRunningTime="2026-11-03 09:06:31.073058462 +0000 UTC m=+1614.741099202" Nov 03 09:06:31.361680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:31.361653 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:31.366924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:31.366905 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:31.953546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:31.953521 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:31.954748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:31.954727 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:06:32.137385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:32.137350 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:32.818628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:32.818541 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:33.153307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:33.153276 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:33.639493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:33.639467 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:33.783379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:33.783345 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:33.920376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:33.920297 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:34.082212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:34.082181 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:34.223352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:34.223266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:34.355754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:34.355673 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:34.852065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:34.852028 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:35.453029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:35.453001 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:35.562645 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:06:35.562610 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:06:37.253735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:37.253703 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:37.465125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:37.465089 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:38.653335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:38.653301 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:38.779365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:38.779310 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:38.926769 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:38.926688 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:40.091435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:40.091403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:42.292647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:42.292610 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:42.489743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:42.489710 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:45.235566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:45.235531 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:46.103378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:46.103346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:49.597267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:49.597238 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:51.447786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:51.447752 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:51.603659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:51.603588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:51.744825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:51.744742 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:53.224960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:53.224933 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:53.743398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:53.743362 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:54.506811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:54.506779 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:56.523068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:56.523035 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:06:56.834443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:06:56.834359 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:00.456726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:00.456699 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:00.587534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:00.587498 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:01.689566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:01.689526 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:02.108204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:02.108106 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:02.270556 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:02.270513 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:02.401562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:02.401530 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:05.562431 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:07:05.562390 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:07:07.777261 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:07.777226 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:08.192402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:08.192356 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:08.266012 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:08.265982 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:07:08.266210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:08.266189 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" containerID="cri-o://7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef" gracePeriod=30 Nov 03 09:07:13.253458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.253423 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:13.535565 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.535542 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:07:13.608256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608228 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608339 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608366 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608483 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608520 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608586 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608561 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608654 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608596 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608654 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.608711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608679 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:07:13.608711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608700 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.608962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608747 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\") pod \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\" (UID: \"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5\") " Nov 03 09:07:13.608962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608843 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.608962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.608856 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.609628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.609602 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.609760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.609633 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.609760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.609675 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.609760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.609692 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.609760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.609706 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.611039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.611012 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "kube-api-access-mp444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:13.611187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.611050 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:07:13.611187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.611071 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:07:13.611187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.611107 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token" pod "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" (UID: "dd2572ad-7c80-42e2-9ea2-7a6e695da0e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:07:13.710800 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.710771 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.710800 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.710800 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp444\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-kube-api-access-mp444\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.710800 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.710810 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:13.711091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:13.710827 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:07:14.222393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.222354 4543 generic.go:356] "Generic (PLEG): container finished" podID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerID="7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef" exitCode=0 Nov 03 09:07:14.222582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.222408 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" Nov 03 09:07:14.222582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.222433 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" event={"ID":"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5","Type":"ContainerDied","Data":"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef"} Nov 03 09:07:14.222582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.222487 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn" event={"ID":"dd2572ad-7c80-42e2-9ea2-7a6e695da0e5","Type":"ContainerDied","Data":"0897455c9a62661a8a6eeb9cf746497efcaa43e842a87ea0003271abd2790dad"} Nov 03 09:07:14.222582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.222513 4543 scope.go:122] "RemoveContainer" containerID="7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef" Nov 03 09:07:14.233346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.233328 4543 scope.go:122] "RemoveContainer" containerID="7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef" Nov 03 09:07:14.233611 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:07:14.233592 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef\": container with ID starting with 7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef not found: ID does not exist" containerID="7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef" Nov 03 09:07:14.233678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.233620 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef"} err="failed to get container status \"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef\": rpc error: code = NotFound desc = could not find container \"7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef\": container with ID starting with 7ba2c229b9a1b40d1a73b44a02fd4759cbf965f9fee6733947941151174942ef not found: ID does not exist" Nov 03 09:07:14.451937 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.451911 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:07:14.497774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:14.497698 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-jbhnn"] Nov 03 09:07:16.452101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:16.452062 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" path="/var/lib/kubelet/pods/dd2572ad-7c80-42e2-9ea2-7a6e695da0e5/volumes" Nov 03 09:07:18.712063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:18.712030 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:24.094238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:24.094202 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:24.285274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:24.285241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:24.413637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:24.413601 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:29.550538 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:29.550506 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:34.410023 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.409989 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:07:34.410758 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.410740 4543 cpu_manager.go:414] "containerMap: removing container" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" Nov 03 09:07:34.410819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.410759 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" Nov 03 09:07:34.410886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.410875 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2572ad-7c80-42e2-9ea2-7a6e695da0e5" containerName="istio-proxy" Nov 03 09:07:34.414587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.414567 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.472019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.471989 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:07:34.502677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.502647 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.502954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.502924 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7q5\" (UniqueName: \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.503051 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.503006 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.503114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.503099 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.518397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.518371 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-04934e-serving-cert\"" Nov 03 09:07:34.599730 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.599699 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:34.604204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.604178 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.604365 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:07:34.604289 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-serving-cert: secret "maas-api-e2e-ipp-a-04934e-serving-cert" not found Nov 03 09:07:34.604365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.604332 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7q5\" (UniqueName: \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.604365 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:07:34.604350 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls podName:457aa574-2c41-41be-bbda-59384f60203b nodeName:}" failed. No retries permitted until 2026-11-03 09:07:35.104330326 +0000 UTC m=+1678.772371043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls") pod "maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" (UID: "457aa574-2c41-41be-bbda-59384f60203b") : secret "maas-api-e2e-ipp-a-04934e-serving-cert" not found Nov 03 09:07:34.604530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.604391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.604530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.604455 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.605042 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.605025 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.606912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.606894 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.683347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.683268 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7q5\" (UniqueName: \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:34.765836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.765799 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:34.945465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:34.945386 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:35.120872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:35.120838 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:35.123476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:35.123447 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:35.330840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:35.330760 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:35.561808 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:07:35.561765 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:07:35.711227 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:07:35.711193 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457aa574_2c41_41be_bbda_59384f60203b.slice/crio-37d8eab536658176914ebe2565016bcd8bc0f3359cde15510283c5f444c55fa7 WatchSource:0}: Error finding container 37d8eab536658176914ebe2565016bcd8bc0f3359cde15510283c5f444c55fa7: Status 404 returned error can't find the container with id 37d8eab536658176914ebe2565016bcd8bc0f3359cde15510283c5f444c55fa7 Nov 03 09:07:35.712107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:35.712081 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:07:36.391684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:36.391653 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerStarted","Data":"37d8eab536658176914ebe2565016bcd8bc0f3359cde15510283c5f444c55fa7"} Nov 03 09:07:37.407635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:37.407394 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerStarted","Data":"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5"} Nov 03 09:07:37.407635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:37.407538 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:37.527743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:37.527690 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" podStartSLOduration=2.713563304 podStartE2EDuration="3.527675223s" totalImagesPullingTime="814.111919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:07:37.524473909 +0000 UTC m=+1681.192514649" watchObservedRunningTime="2026-11-03 09:07:37.527675223 +0000 UTC m=+1681.195715962" Nov 03 09:07:38.994559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:38.994505 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:39.832418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:39.832355 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:43.415609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:43.415560 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.77:8443/health\": dial tcp 10.134.0.77:8443: connect: connection refused" Nov 03 09:07:44.956969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:44.956916 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:46.448752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:46.448717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:47.470395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:47.470361 4543 generic.go:356] "Generic (PLEG): container finished" podID="457aa574-2c41-41be-bbda-59384f60203b" containerID="52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5" exitCode=1 Nov 03 09:07:47.470815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:47.470439 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerDied","Data":"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5"} Nov 03 09:07:47.471010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:47.470994 4543 scope.go:122] "RemoveContainer" containerID="52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5" Nov 03 09:07:48.479129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:48.479095 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerStarted","Data":"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a"} Nov 03 09:07:48.479528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:48.479328 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:50.026080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:50.026050 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:54.441198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:54.441134 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:54.492076 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:54.492049 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:07:55.169470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:55.169439 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:07:56.254718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:07:56.254692 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:01.534362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:01.534329 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:01.663978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:01.663944 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:03.152932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:03.152898 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:03.650886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:03.650846 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:05.561445 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:05.561413 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:08:06.194592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.194560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:06.597685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.597588 4543 generic.go:356] "Generic (PLEG): container finished" podID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerID="0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2" exitCode=0 Nov 03 09:08:06.597685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.597634 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerDied","Data":"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2"} Nov 03 09:08:06.598248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.598026 4543 scope.go:122] "RemoveContainer" containerID="0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2" Nov 03 09:08:06.599809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.599770 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:06.599903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.599864 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:06.599903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:06.599897 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:07.019120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:07.019086 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:07.612024 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:07.611986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerStarted","Data":"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9"} Nov 03 09:08:08.149701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:08.149642 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:08.362443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:08.362409 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:08:08.362656 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:08.362636 4543 patch_prober.go:29] interesting pod/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 09:08:08.362722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:08.362690 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 09:08:09.362433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:09.362399 4543 patch_prober.go:29] interesting pod/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 09:08:09.362848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:09.362450 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 09:08:10.362705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:10.362675 4543 patch_prober.go:29] interesting pod/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 09:08:10.363122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:10.362726 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 09:08:11.362787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:11.362750 4543 patch_prober.go:29] interesting pod/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 09:08:11.363243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:11.362803 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 09:08:11.671628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:11.671542 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:08:11.671834 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:11.671812 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" podUID="496d821e-6c79-473b-b7c7-fa54c575ae39" containerName="maas-api" containerID="cri-o://f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1" gracePeriod=30 Nov 03 09:08:11.762721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:11.762695 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:12.062211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.061222 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:08:12.223169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.219730 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\") pod \"496d821e-6c79-473b-b7c7-fa54c575ae39\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " Nov 03 09:08:12.223169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.219919 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\" (UniqueName: \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\") pod \"496d821e-6c79-473b-b7c7-fa54c575ae39\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " Nov 03 09:08:12.223169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.219991 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") pod \"496d821e-6c79-473b-b7c7-fa54c575ae39\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " Nov 03 09:08:12.223169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.220080 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\") pod \"496d821e-6c79-473b-b7c7-fa54c575ae39\" (UID: \"496d821e-6c79-473b-b7c7-fa54c575ae39\") " Nov 03 09:08:12.245827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.231704 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs" pod "496d821e-6c79-473b-b7c7-fa54c575ae39" (UID: "496d821e-6c79-473b-b7c7-fa54c575ae39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:08:12.256094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.250962 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca" pod "496d821e-6c79-473b-b7c7-fa54c575ae39" (UID: "496d821e-6c79-473b-b7c7-fa54c575ae39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:12.256094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.252818 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr" pod "496d821e-6c79-473b-b7c7-fa54c575ae39" (UID: "496d821e-6c79-473b-b7c7-fa54c575ae39"). InnerVolumeSpecName "kube-api-access-h4qdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:12.279328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.256615 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls" pod "496d821e-6c79-473b-b7c7-fa54c575ae39" (UID: "496d821e-6c79-473b-b7c7-fa54c575ae39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:08:12.335482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.334515 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4qdr\" (UniqueName: \"kubernetes.io/projected/496d821e-6c79-473b-b7c7-fa54c575ae39-kube-api-access-h4qdr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:12.335482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.334548 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:12.335482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.334564 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496d821e-6c79-473b-b7c7-fa54c575ae39-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:12.335482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.334578 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/496d821e-6c79-473b-b7c7-fa54c575ae39-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:12.362071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.362034 4543 patch_prober.go:29] interesting pod/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 09:08:12.362243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.362092 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 09:08:12.645117 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.645083 4543 generic.go:356] "Generic (PLEG): container finished" podID="496d821e-6c79-473b-b7c7-fa54c575ae39" containerID="f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1" exitCode=0 Nov 03 09:08:12.645572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.645132 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" Nov 03 09:08:12.645572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.645177 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" event={"ID":"496d821e-6c79-473b-b7c7-fa54c575ae39","Type":"ContainerDied","Data":"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1"} Nov 03 09:08:12.645572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.645231 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5" event={"ID":"496d821e-6c79-473b-b7c7-fa54c575ae39","Type":"ContainerDied","Data":"0608c568aa44342e17d9353f0d64138e2b9e32b2afe0a33d8777ed31ef90171b"} Nov 03 09:08:12.645572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.645252 4543 scope.go:122] "RemoveContainer" containerID="f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1" Nov 03 09:08:12.655309 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.655285 4543 scope.go:122] "RemoveContainer" containerID="f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1" Nov 03 09:08:12.655615 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:12.655585 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1\": container with ID starting with f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1 not found: ID does not exist" containerID="f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1" Nov 03 09:08:12.655745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.655624 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1"} err="failed to get container status \"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1\": rpc error: code = NotFound desc = could not find container \"f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1\": container with ID starting with f17dba4a505b5b125a0d80996bbf50bc39c0b191c8f4686216fabbe6f2312ba1 not found: ID does not exist" Nov 03 09:08:12.817726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.817697 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:08:12.824121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:12.824088 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-npbr5"] Nov 03 09:08:13.366806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.366777 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:08:13.367186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.367167 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:08:13.367958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.367940 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:08:13.462245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.462215 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:13.732425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.732349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:13.866903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:13.866802 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:14.452277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:14.452247 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496d821e-6c79-473b-b7c7-fa54c575ae39" path="/var/lib/kubelet/pods/496d821e-6c79-473b-b7c7-fa54c575ae39/volumes" Nov 03 09:08:14.662451 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:14.662418 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:14.797695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:14.797620 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:15.697941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:15.697911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:17.023404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:17.023342 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:17.716183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:17.711408 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:17.854687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:17.854646 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:17.987551 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:17.987472 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:19.550912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:19.550882 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:21.854412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:21.854360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:22.069385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:22.069345 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:22.210287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:22.210247 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:22.489941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:22.489818 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:25.038747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:25.038703 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:25.170728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:25.170688 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:25.311795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:25.311698 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:25.750038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:25.749986 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:26.439318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.439286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:26.987362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.987332 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:08:26.987912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.987899 4543 cpu_manager.go:414] "containerMap: removing container" podUID="496d821e-6c79-473b-b7c7-fa54c575ae39" containerName="maas-api" Nov 03 09:08:26.987972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.987913 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496d821e-6c79-473b-b7c7-fa54c575ae39" containerName="maas-api" Nov 03 09:08:26.988038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.988029 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="496d821e-6c79-473b-b7c7-fa54c575ae39" containerName="maas-api" Nov 03 09:08:26.991013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:26.990990 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132234 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132285 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132324 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132358 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132406 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132460 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7lt\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132531 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132534 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:27.132705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132694 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.133352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.132812 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.150197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.150167 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bcc5ce-openshift-default-dockercfg-nw72p\"" Nov 03 09:08:27.154927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.154893 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:08:27.234128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234094 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234271 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234297 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234335 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234388 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234440 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7lt\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234517 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234598 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.234743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.234642 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.235089 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.235065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.235197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.235065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.235343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.235322 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.235413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.235330 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.238047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.237983 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.239276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.239222 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.295130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.295099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.295322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.295261 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7lt\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\") pod \"e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.304034 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.304006 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:27.428279 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.425711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:27.759760 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:08:27.759733 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec19697_784d_49f5_833b_f1e146f77627.slice/crio-e789e4da5d878554087149f61938d2eb07597cc50d80599feee2ce8fda5a9bf4 WatchSource:0}: Error finding container e789e4da5d878554087149f61938d2eb07597cc50d80599feee2ce8fda5a9bf4: Status 404 returned error can't find the container with id e789e4da5d878554087149f61938d2eb07597cc50d80599feee2ce8fda5a9bf4 Nov 03 09:08:27.760119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.759946 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:08:27.762089 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.762059 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:27.762212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.762120 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:27.762212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:27.762175 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:08:28.770210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:28.770170 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerStarted","Data":"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9"} Nov 03 09:08:28.770616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:28.770231 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerStarted","Data":"e789e4da5d878554087149f61938d2eb07597cc50d80599feee2ce8fda5a9bf4"} Nov 03 09:08:28.994398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:28.994347 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podStartSLOduration=2.994333479 podStartE2EDuration="2.994333479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:28.942385199 +0000 UTC m=+1732.610425940" watchObservedRunningTime="2026-11-03 09:08:28.994333479 +0000 UTC m=+1732.662374218" Nov 03 09:08:29.304387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:29.304349 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:29.309942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:29.309916 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:29.777210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:29.777179 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:29.778283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:29.778263 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:08:30.726743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:30.726711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:33.502291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:33.502258 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:33.704152 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:33.704101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:34.859430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.859396 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:08:34.859875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.859601 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" containerID="cri-o://147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12" gracePeriod=30 Nov 03 09:08:34.909664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.909633 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 09:08:34.909853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.909832 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" containerID="cri-o://612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b" gracePeriod=30 Nov 03 09:08:34.913041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.913016 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:08:34.913235 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.913216 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" containerID="cri-o://77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f" gracePeriod=30 Nov 03 09:08:34.962061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.962029 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:08:34.962276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:34.962252 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" containerID="cri-o://29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87" gracePeriod=30 Nov 03 09:08:35.639967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:35.638392 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:35.708353 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:35.708320 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:08:35.885845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:35.885797 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:36.067528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:36.067443 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:37.889266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:37.889234 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:38.718698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:38.718656 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:08:38.718862 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:38.718722 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:08:39.249503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:39.249462 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:39.387349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:39.387314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:39.938508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:39.938442 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:40.222182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.219585 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:08:40.298653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.298626 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 09:08:40.311754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.311730 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:08:40.348314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.348294 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:08:40.373751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373730 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.373876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373771 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.373876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373842 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.373876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373863 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373932 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.373989 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374036 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374077 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374110 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374112 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.374260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374175 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374245 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374271 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374314 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.374512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374327 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.374512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374334 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:40.374512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374324 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374525 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374561 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374624 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374671 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\") pod \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\" (UID: \"939cd6ec-76fd-400b-a6e9-5f57fe037a7f\") " Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374676 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374725 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.374939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374775 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\") pod \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\" (UID: \"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb\") " Nov 03 09:08:40.375325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.374965 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.375325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.375279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.375435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.375322 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:40.375638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.375606 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.376082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376057 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376090 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376108 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376123 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376156 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376170 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376183 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376197 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376210 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376224 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.376910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.376885 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.377105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377079 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:40.377310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377286 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "kube-api-access-8d4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.377495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377471 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "kube-api-access-fhkf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.377698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.377919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377894 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.378001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.377976 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo" pod "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" (UID: "b467afe3-a4dd-48e0-b8bb-034efa1aa4cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:40.378072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.378055 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token" pod "939cd6ec-76fd-400b-a6e9-5f57fe037a7f" (UID: "939cd6ec-76fd-400b-a6e9-5f57fe037a7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.477330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477305 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477349 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477480 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477504 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477539 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477565 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477569 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.477646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477613 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477643 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477668 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477696 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477745 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477806 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477860 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477859 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477898 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.477906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477905 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\") pod \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\" (UID: \"8ea552a0-00d9-400d-8dd2-958d1df54ca5\") " Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477933 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.477986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478033 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478125 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478132 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478209 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\") pod \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\" (UID: \"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511\") " Nov 03 09:08:40.478420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.478779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478879 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478900 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478916 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4mv\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-kube-api-access-8d4mv\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478930 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478944 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478957 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478970 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478985 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.478998 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479012 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479027 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479041 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479054 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479069 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479083 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479096 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhkf7\" (UniqueName: \"kubernetes.io/projected/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-kube-api-access-fhkf7\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479111 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.479492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.479124 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/939cd6ec-76fd-400b-a6e9-5f57fe037a7f-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.480679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.480629 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "kube-api-access-w4rpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.480821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.480784 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.480918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.480821 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "kube-api-access-7x2dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.480918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.480892 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.481015 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.480942 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:40.481168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.481119 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo" pod "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" (UID: "39a7ea5e-d888-4edb-9c00-1cd3ee6f7511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:40.482016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.481997 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:40.482168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.482120 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token" pod "8ea552a0-00d9-400d-8dd2-958d1df54ca5" (UID: "8ea552a0-00d9-400d-8dd2-958d1df54ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:40.580111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580078 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580111 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580125 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580176 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580192 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x2dw\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-kube-api-access-7x2dw\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580208 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea552a0-00d9-400d-8dd2-958d1df54ca5-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580221 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.580355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.580236 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rpd\" (UniqueName: \"kubernetes.io/projected/8ea552a0-00d9-400d-8dd2-958d1df54ca5-kube-api-access-w4rpd\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:40.752281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.752198 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:40.860249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.860218 4543 generic.go:356] "Generic (PLEG): container finished" podID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerID="147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12" exitCode=0 Nov 03 09:08:40.860432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.860283 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" Nov 03 09:08:40.860432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.860313 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" event={"ID":"939cd6ec-76fd-400b-a6e9-5f57fe037a7f","Type":"ContainerDied","Data":"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12"} Nov 03 09:08:40.860432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.860393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2" event={"ID":"939cd6ec-76fd-400b-a6e9-5f57fe037a7f","Type":"ContainerDied","Data":"9e8faee225575b9efc31e24f3ca2e33ebf1a8893ab312c4a269595bdd2039224"} Nov 03 09:08:40.860432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.860417 4543 scope.go:122] "RemoveContainer" containerID="147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12" Nov 03 09:08:40.861982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.861957 4543 generic.go:356] "Generic (PLEG): container finished" podID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerID="29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87" exitCode=0 Nov 03 09:08:40.862313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.862022 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" Nov 03 09:08:40.862313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.862037 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" event={"ID":"8ea552a0-00d9-400d-8dd2-958d1df54ca5","Type":"ContainerDied","Data":"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87"} Nov 03 09:08:40.862313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.862075 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74" event={"ID":"8ea552a0-00d9-400d-8dd2-958d1df54ca5","Type":"ContainerDied","Data":"8271d2db6f07a35fb43cb262771033caace2e8f8b85a85b8c39c232bd75e0705"} Nov 03 09:08:40.865886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.865861 4543 generic.go:356] "Generic (PLEG): container finished" podID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerID="77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f" exitCode=0 Nov 03 09:08:40.866596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.866350 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" Nov 03 09:08:40.869806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.868891 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" event={"ID":"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511","Type":"ContainerDied","Data":"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f"} Nov 03 09:08:40.869806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.868948 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r" event={"ID":"39a7ea5e-d888-4edb-9c00-1cd3ee6f7511","Type":"ContainerDied","Data":"1a04584829ee22b60cc09636987b5ea7c34c570eaf389eebe03be6b9c7ab0670"} Nov 03 09:08:40.870947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.870925 4543 generic.go:356] "Generic (PLEG): container finished" podID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerID="612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b" exitCode=0 Nov 03 09:08:40.871094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.870963 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" event={"ID":"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb","Type":"ContainerDied","Data":"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b"} Nov 03 09:08:40.871094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.870976 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" Nov 03 09:08:40.871094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.871000 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd" event={"ID":"b467afe3-a4dd-48e0-b8bb-034efa1aa4cb","Type":"ContainerDied","Data":"2f5dab0489caad6050fbad41026eb20d535695dcd675213ea90e6cbba4fd9c15"} Nov 03 09:08:40.875039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.874965 4543 scope.go:122] "RemoveContainer" containerID="147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12" Nov 03 09:08:40.875480 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:40.875457 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12\": container with ID starting with 147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12 not found: ID does not exist" containerID="147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12" Nov 03 09:08:40.875570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.875488 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12"} err="failed to get container status \"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12\": rpc error: code = NotFound desc = could not find container \"147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12\": container with ID starting with 147b7eecff749735f0b6466e13fb62f48c562abe12dc8a602764e666ca57cc12 not found: ID does not exist" Nov 03 09:08:40.875570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.875505 4543 scope.go:122] "RemoveContainer" containerID="29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87" Nov 03 09:08:40.919618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.919594 4543 scope.go:122] "RemoveContainer" containerID="29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87" Nov 03 09:08:40.919963 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:40.919942 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87\": container with ID starting with 29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87 not found: ID does not exist" containerID="29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87" Nov 03 09:08:40.920035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.919971 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87"} err="failed to get container status \"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87\": rpc error: code = NotFound desc = could not find container \"29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87\": container with ID starting with 29e23bab3723b3bdbed63e75dac057dac8846fdd91fecae01062ebb770dd4b87 not found: ID does not exist" Nov 03 09:08:40.920035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.919989 4543 scope.go:122] "RemoveContainer" containerID="77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f" Nov 03 09:08:40.929105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.929082 4543 scope.go:122] "RemoveContainer" containerID="77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f" Nov 03 09:08:40.929402 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:40.929384 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f\": container with ID starting with 77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f not found: ID does not exist" containerID="77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f" Nov 03 09:08:40.929463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.929409 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f"} err="failed to get container status \"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f\": rpc error: code = NotFound desc = could not find container \"77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f\": container with ID starting with 77c3fdc2856115844567bd895037bdfaebdf16014961a20462c9e40d7e90d76f not found: ID does not exist" Nov 03 09:08:40.929463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.929425 4543 scope.go:122] "RemoveContainer" containerID="612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b" Nov 03 09:08:40.938226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.938211 4543 scope.go:122] "RemoveContainer" containerID="612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b" Nov 03 09:08:40.938510 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:40.938493 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b\": container with ID starting with 612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b not found: ID does not exist" containerID="612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b" Nov 03 09:08:40.938555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:40.938519 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b"} err="failed to get container status \"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b\": rpc error: code = NotFound desc = could not find container \"612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b\": container with ID starting with 612c5bf6b1a3d00814938c9342ba50245e9d7ae0cb7f306df34f9f2929c89c9b not found: ID does not exist" Nov 03 09:08:41.032272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.032198 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 09:08:41.039789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.039764 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-zccmd"] Nov 03 09:08:41.153127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.153088 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:08:41.159302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.159275 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-cj4j2"] Nov 03 09:08:41.273271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.273241 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:08:41.278232 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.278204 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6fq74"] Nov 03 09:08:41.391239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.391210 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:08:41.397053 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.397022 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wjm7r"] Nov 03 09:08:41.834995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.834911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:41.982059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:41.982031 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:42.119388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.119358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:42.264307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.264278 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:42.392403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.392330 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:42.455187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.455130 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" path="/var/lib/kubelet/pods/39a7ea5e-d888-4edb-9c00-1cd3ee6f7511/volumes" Nov 03 09:08:42.455651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.455636 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" path="/var/lib/kubelet/pods/8ea552a0-00d9-400d-8dd2-958d1df54ca5/volumes" Nov 03 09:08:42.456032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.456019 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" path="/var/lib/kubelet/pods/939cd6ec-76fd-400b-a6e9-5f57fe037a7f/volumes" Nov 03 09:08:42.457119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:42.457095 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" path="/var/lib/kubelet/pods/b467afe3-a4dd-48e0-b8bb-034efa1aa4cb/volumes" Nov 03 09:08:43.709803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:43.709756 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:45.909425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:45.909395 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:46.102186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:46.102133 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:46.843077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:46.843005 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:47.757186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:47.757155 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:47.944198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:47.944132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:48.184958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:48.180751 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:49.580774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:49.580747 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:51.057071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:51.057036 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:51.240070 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:51.240035 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:52.945465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:52.945430 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:53.463572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:53.463531 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:53.479491 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:53.479466 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:08:53.479683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:53.479662 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" containerID="cri-o://178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69" gracePeriod=30 Nov 03 09:08:54.234631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:54.234596 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:55.294362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:55.294318 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:08:58.749837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.749778 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:08:58.864453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864422 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864469 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864525 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864555 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864610 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864632 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864662 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864727 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:58.864925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864767 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864800 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\") pod \"f483ea39-8bfa-424a-93c2-34e46689ff00\" (UID: \"f483ea39-8bfa-424a-93c2-34e46689ff00\") " Nov 03 09:08:58.864925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864822 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:58.865230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.864923 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:58.865230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865032 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865269 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865476 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865492 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865501 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865509 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.865516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.865518 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f483ea39-8bfa-424a-93c2-34e46689ff00-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.867011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.866982 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "kube-api-access-znscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:58.867180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.867066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:08:58.867416 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.867398 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:08:58.867473 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.867412 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token" pod "f483ea39-8bfa-424a-93c2-34e46689ff00" (UID: "f483ea39-8bfa-424a-93c2-34e46689ff00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:08:58.967079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.967044 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znscm\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-kube-api-access-znscm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.967079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.967074 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.967079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.967083 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.967330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.967092 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f483ea39-8bfa-424a-93c2-34e46689ff00-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:08:58.988212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.988181 4543 generic.go:356] "Generic (PLEG): container finished" podID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerID="178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69" exitCode=0 Nov 03 09:08:58.988370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.988238 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" Nov 03 09:08:58.988370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.988236 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" event={"ID":"f483ea39-8bfa-424a-93c2-34e46689ff00","Type":"ContainerDied","Data":"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69"} Nov 03 09:08:58.988488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.988374 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz" event={"ID":"f483ea39-8bfa-424a-93c2-34e46689ff00","Type":"ContainerDied","Data":"bef6c50f3467bbc81df0cd20a6df0e1f1156837f6fffec9abb15a3dc3a6525ea"} Nov 03 09:08:58.988488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.988416 4543 scope.go:122] "RemoveContainer" containerID="178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69" Nov 03 09:08:58.999372 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.999354 4543 scope.go:122] "RemoveContainer" containerID="178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69" Nov 03 09:08:58.999658 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:08:58.999639 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69\": container with ID starting with 178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69 not found: ID does not exist" containerID="178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69" Nov 03 09:08:58.999700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:58.999667 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69"} err="failed to get container status \"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69\": rpc error: code = NotFound desc = could not find container \"178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69\": container with ID starting with 178f37ed101acd9c2cb28d4cb262c5d5fb824e9c697ac5a7b4484622d9b8fd69 not found: ID does not exist" Nov 03 09:08:59.159971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:59.159940 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:08:59.168799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:08:59.168761 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-nrnnz"] Nov 03 09:09:00.452219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:00.452186 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" path="/var/lib/kubelet/pods/f483ea39-8bfa-424a-93c2-34e46689ff00/volumes" Nov 03 09:09:00.702430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:00.702348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:00.832470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:00.832424 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:01.249884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:01.249849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:01.392299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:01.392270 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:05.561563 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:05.561527 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:09:05.692084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692052 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:09:05.692690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692675 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692691 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692706 4543 cpu_manager.go:414] "containerMap: removing container" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692711 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692717 4543 cpu_manager.go:414] "containerMap: removing container" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692722 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692737 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" Nov 03 09:09:05.692739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692741 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692759 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692764 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692850 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b467afe3-a4dd-48e0-b8bb-034efa1aa4cb" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692861 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="939cd6ec-76fd-400b-a6e9-5f57fe037a7f" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692868 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a7ea5e-d888-4edb-9c00-1cd3ee6f7511" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692875 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f483ea39-8bfa-424a-93c2-34e46689ff00" containerName="istio-proxy" Nov 03 09:09:05.692947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.692882 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea552a0-00d9-400d-8dd2-958d1df54ca5" containerName="istio-proxy" Nov 03 09:09:05.697745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.697728 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.754738 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.754707 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:09:05.800645 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.800617 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b0c653f4-serving-cert\"" Nov 03 09:09:05.858787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.858716 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnnnt\" (UniqueName: \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.858936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.858918 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.858983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.858959 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.859075 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.859063 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.908047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.908012 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.960543 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnnnt\" (UniqueName: \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.960755 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.960801 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.960862 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:05.961063 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-serving-cert: secret "maas-api-e2e-mt-b0c653f4-serving-cert" not found Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:05.961159 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls podName:32135f10-d3de-4a0d-896a-7b47658f9b21 nodeName:}" failed. No retries permitted until 2026-11-03 09:09:06.461119222 +0000 UTC m=+1770.129159957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls") pod "maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" (UID: "32135f10-d3de-4a0d-896a-7b47658f9b21") : secret "maas-api-e2e-mt-b0c653f4-serving-cert" not found Nov 03 09:09:05.962245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.961944 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:05.975873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:05.975836 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:06.020496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.020465 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnnnt\" (UniqueName: \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:06.047576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.047535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:06.203761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.203724 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:06.468583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.468509 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:06.471030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.471008 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") pod \"maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:06.611638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.611594 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:06.866745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.866716 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:06.992772 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:09:06.992743 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32135f10_d3de_4a0d_896a_7b47658f9b21.slice/crio-24d576777a19bcadc29940727aa425202193aa62bf06d883c5c72f1da34d63b0 WatchSource:0}: Error finding container 24d576777a19bcadc29940727aa425202193aa62bf06d883c5c72f1da34d63b0: Status 404 returned error can't find the container with id 24d576777a19bcadc29940727aa425202193aa62bf06d883c5c72f1da34d63b0 Nov 03 09:09:06.993168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.993123 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:09:06.994270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.994242 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:06.994379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:06.994315 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:09:07.045208 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:07.045128 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" event={"ID":"32135f10-d3de-4a0d-896a-7b47658f9b21","Type":"ContainerStarted","Data":"24d576777a19bcadc29940727aa425202193aa62bf06d883c5c72f1da34d63b0"} Nov 03 09:09:07.659940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:07.659910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:08.055444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:08.055411 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" event={"ID":"32135f10-d3de-4a0d-896a-7b47658f9b21","Type":"ContainerStarted","Data":"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968"} Nov 03 09:09:08.055636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:08.055548 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:08.220611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:08.214676 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" podStartSLOduration=2.3419187089999998 podStartE2EDuration="3.214657496s" totalImagesPullingTime="872.738787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:08.17309279 +0000 UTC m=+1771.841133529" watchObservedRunningTime="2026-11-03 09:09:08.214657496 +0000 UTC m=+1771.882698237" Nov 03 09:09:09.011804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:09.011771 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:11.083423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:11.083390 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:14.068722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:14.068696 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:09:16.036233 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:16.036200 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:20.972383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:20.972350 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:21.139252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:21.139216 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:21.282085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:21.281987 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:24.237644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:24.237607 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:26.162035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:26.161999 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:31.400373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:31.400338 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:35.022212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:35.021999 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:35.150495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:35.150461 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:35.561964 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:35.561930 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:09:35.644032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:35.644002 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:36.654273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.654238 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:09:36.658574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.658549 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:09:36.757030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.757003 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:09:36.762405 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.762388 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:36.818868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.818836 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:09:36.865242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.865212 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04934e\"" Nov 03 09:09:36.865415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.865254 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04934e-dockercfg-xg4kf\"" Nov 03 09:09:36.945917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.945842 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtg9\" (UniqueName: \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:36.946061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:36.946046 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.046832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.046791 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.047028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.046888 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtg9\" (UniqueName: \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.047641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.047612 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.106838 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.106805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtg9\" (UniqueName: \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\") pod \"payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.377635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.377608 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:37.758606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:37.758576 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:09:37.759417 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:09:37.759394 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f97bab_2199_43e5_acca_1181cfb09ce9.slice/crio-b5a3c47a43d778b7317bc4368be1a012aabc13b8e9bae4c94223a66c1485dbc9 WatchSource:0}: Error finding container b5a3c47a43d778b7317bc4368be1a012aabc13b8e9bae4c94223a66c1485dbc9: Status 404 returned error can't find the container with id b5a3c47a43d778b7317bc4368be1a012aabc13b8e9bae4c94223a66c1485dbc9 Nov 03 09:09:38.259497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:38.259463 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" event={"ID":"d4f97bab-2199-43e5-acca-1181cfb09ce9","Type":"ContainerStarted","Data":"b5a3c47a43d778b7317bc4368be1a012aabc13b8e9bae4c94223a66c1485dbc9"} Nov 03 09:09:38.347335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:38.347298 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:39.681577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:39.681490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:40.273448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:40.273414 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" event={"ID":"d4f97bab-2199-43e5-acca-1181cfb09ce9","Type":"ContainerStarted","Data":"5e03022165e291142cfd45610c65b99aacd08a9155377d001aa15a2f6b6bf1a8"} Nov 03 09:09:40.273615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:40.273500 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:40.392112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:40.392056 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" podStartSLOduration=2.752119811 podStartE2EDuration="4.392042496s" totalImagesPullingTime="1.639922685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:40.390323073 +0000 UTC m=+1804.058363813" watchObservedRunningTime="2026-11-03 09:09:40.392042496 +0000 UTC m=+1804.060083235" Nov 03 09:09:40.461216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:40.461181 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:43.280997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:43.280965 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:43.443581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:43.443546 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:43.590263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:43.590172 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:47.202834 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:47.202801 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:48.544346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:48.544317 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:50.484234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:50.484199 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:09:50.484670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:50.484422 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" containerID="cri-o://3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7" gracePeriod=30 Nov 03 09:09:50.534467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:50.534437 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:09:50.534721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:50.534675 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" containerID="cri-o://3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158" gracePeriod=30 Nov 03 09:09:51.281374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:51.281346 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:09:53.292567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.292537 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:09:53.297184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.297167 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.354071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.354037 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:09:53.400723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.400688 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c32d41-serving-cert\"" Nov 03 09:09:53.480099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.480071 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.480321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.480119 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.480321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.480254 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.480321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.480279 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696z4\" (UniqueName: \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.487802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.487775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:53.582165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.582064 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.582358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.582265 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.582474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.582455 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.582541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.582489 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696z4\" (UniqueName: \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.582928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.582859 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.585791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.585760 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.585924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.585799 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.641986 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.641960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696z4\" (UniqueName: \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\") pod \"maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:53.653053 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.653023 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:53.783462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.783431 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:53.911103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:53.911070 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:54.290718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:54.290688 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:09:54.291309 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:09:54.291275 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5c36fb_82d4_4a22_ab23_2dced0f945e8.slice/crio-9820c8e4043ab06effa5739ad37eb5b446ac63ada9b48ca521f3cbcd8504ae8b WatchSource:0}: Error finding container 9820c8e4043ab06effa5739ad37eb5b446ac63ada9b48ca521f3cbcd8504ae8b: Status 404 returned error can't find the container with id 9820c8e4043ab06effa5739ad37eb5b446ac63ada9b48ca521f3cbcd8504ae8b Nov 03 09:09:54.369552 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:54.369519 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" event={"ID":"6b5c36fb-82d4-4a22-ab23-2dced0f945e8","Type":"ContainerStarted","Data":"9820c8e4043ab06effa5739ad37eb5b446ac63ada9b48ca521f3cbcd8504ae8b"} Nov 03 09:09:55.377223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.377186 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" event={"ID":"6b5c36fb-82d4-4a22-ab23-2dced0f945e8","Type":"ContainerStarted","Data":"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2"} Nov 03 09:09:55.377619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.377253 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:09:55.497839 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.497773 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" podStartSLOduration=1.622832522 podStartE2EDuration="2.497757913s" totalImagesPullingTime="874.925391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:55.494199589 +0000 UTC m=+1819.162240330" watchObservedRunningTime="2026-11-03 09:09:55.497757913 +0000 UTC m=+1819.165798651" Nov 03 09:09:55.811951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.811921 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:09:55.854413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.854387 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:09:55.914670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914645 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.914811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914763 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.914855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914831 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.914889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914870 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:55.914889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914882 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.914956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914915 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.915006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.914990 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.915049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915024 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.915104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915083 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.915193 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915103 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:55.915278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915253 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:55.915336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915303 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\") pod \"b417ea79-970a-41cc-93df-d6547cf8c414\" (UID: \"b417ea79-970a-41cc-93df-d6547cf8c414\") " Nov 03 09:09:55.915433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915309 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:55.915433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.915325 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:55.916065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.916045 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:55.916168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.916074 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:55.916168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.916088 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b417ea79-970a-41cc-93df-d6547cf8c414-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:55.916168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.916101 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:55.916168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.916114 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:55.917326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.917298 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:55.917552 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.917533 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "kube-api-access-wsm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:55.917777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.917756 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:09:55.917777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:55.917768 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy" pod "b417ea79-970a-41cc-93df-d6547cf8c414" (UID: "b417ea79-970a-41cc-93df-d6547cf8c414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.017475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017448 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017513 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017603 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017636 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017684 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017724 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017781 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017810 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.017835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017744 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.018155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017858 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\") pod \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\" (UID: \"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70\") " Nov 03 09:09:56.018155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.017919 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.018155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018030 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.018385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018352 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:09:56.018676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018474 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018496 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b417ea79-970a-41cc-93df-d6547cf8c414-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018510 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018540 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b417ea79-970a-41cc-93df-d6547cf8c414-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018584 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018694 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018739 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsm95\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-kube-api-access-wsm95\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018754 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.018965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.018767 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b417ea79-970a-41cc-93df-d6547cf8c414-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.020079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.020057 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "kube-api-access-fjdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:56.020194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.020083 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:09:56.020194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.020096 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:09:56.020194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.020106 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token" pod "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" (UID: "ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:09:56.120100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.120068 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.120100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.120094 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjdtg\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-kube-api-access-fjdtg\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.120100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.120105 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.120361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.120113 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:09:56.385058 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.384963 4543 generic.go:356] "Generic (PLEG): container finished" podID="b417ea79-970a-41cc-93df-d6547cf8c414" containerID="3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158" exitCode=0 Nov 03 09:09:56.385058 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.385016 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" Nov 03 09:09:56.385622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.385047 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" event={"ID":"b417ea79-970a-41cc-93df-d6547cf8c414","Type":"ContainerDied","Data":"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158"} Nov 03 09:09:56.385622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.385113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw" event={"ID":"b417ea79-970a-41cc-93df-d6547cf8c414","Type":"ContainerDied","Data":"9bc5176a96caa6bc31bd0a88df94dfc7deff65d6697b6aaec7b5ff5cc001f196"} Nov 03 09:09:56.385622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.385159 4543 scope.go:122] "RemoveContainer" containerID="3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158" Nov 03 09:09:56.386925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.386898 4543 generic.go:356] "Generic (PLEG): container finished" podID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerID="3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7" exitCode=0 Nov 03 09:09:56.387693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.387677 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" Nov 03 09:09:56.389580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.388917 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" event={"ID":"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70","Type":"ContainerDied","Data":"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7"} Nov 03 09:09:56.389580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.388968 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t" event={"ID":"ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70","Type":"ContainerDied","Data":"c7b37f20cef68dd9ad4b69d01ed027e0b974da06c7e91a7f579a331445dc8126"} Nov 03 09:09:56.397044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.397018 4543 scope.go:122] "RemoveContainer" containerID="3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158" Nov 03 09:09:56.397348 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:56.397328 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158\": container with ID starting with 3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158 not found: ID does not exist" containerID="3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158" Nov 03 09:09:56.397427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.397357 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158"} err="failed to get container status \"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158\": rpc error: code = NotFound desc = could not find container \"3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158\": container with ID starting with 3a99283321bd2edfb65a486946c4d8ddb519f8632d291856f472e297304ee158 not found: ID does not exist" Nov 03 09:09:56.397427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.397375 4543 scope.go:122] "RemoveContainer" containerID="3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7" Nov 03 09:09:56.407453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.407433 4543 scope.go:122] "RemoveContainer" containerID="3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7" Nov 03 09:09:56.407735 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:09:56.407715 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7\": container with ID starting with 3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7 not found: ID does not exist" containerID="3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7" Nov 03 09:09:56.407780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.407744 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7"} err="failed to get container status \"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7\": rpc error: code = NotFound desc = could not find container \"3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7\": container with ID starting with 3253d3b18177ea5da66cbd78d3ba3d33a28844a1b60223d947ef714ed7fa1aa7 not found: ID does not exist" Nov 03 09:09:56.558602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.558566 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:09:56.563262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.563237 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-bfslw"] Nov 03 09:09:56.674238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.674159 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:09:56.680757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:56.680729 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-2b78t"] Nov 03 09:09:58.073572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:58.073540 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:09:58.468525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:58.468483 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" path="/var/lib/kubelet/pods/b417ea79-970a-41cc-93df-d6547cf8c414/volumes" Nov 03 09:09:58.469092 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:58.469069 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" path="/var/lib/kubelet/pods/ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70/volumes" Nov 03 09:09:59.975450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:09:59.975419 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:01.254205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:01.254171 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:01.393471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:01.393442 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:10:01.398629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:01.398597 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:02.952646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:02.952612 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:03.500802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:03.500768 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:04.388588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.388555 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:04.929427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.929389 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:10:04.930025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930005 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" Nov 03 09:10:04.930130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930027 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" Nov 03 09:10:04.930130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930069 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" Nov 03 09:10:04.930130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930076 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" Nov 03 09:10:04.930317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930271 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0b6c34-6e16-4db3-a884-f9c3c3c8ee70" containerName="istio-proxy" Nov 03 09:10:04.930317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.930295 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b417ea79-970a-41cc-93df-d6547cf8c414" containerName="istio-proxy" Nov 03 09:10:04.934840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:04.934820 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.036011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.035937 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:05.049974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.047091 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-77da2a-openshift-default-dockercfg-vggnn\"" Nov 03 09:10:05.051084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.051047 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053381 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053433 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053498 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv27z\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053653 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053680 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053764 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053793 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.055081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.053832 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.154786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.154749 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.154988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.154793 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.154988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.154834 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.154988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.154867 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.154988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.154894 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155017 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155047 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155104 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv27z\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155166 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155253 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155467 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155513 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155585 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.155849 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.155807 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.158472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.158446 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.158666 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.158649 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.217648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.217569 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv27z\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.225710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.225681 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\") pod \"e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.249246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.249214 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:05.452027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.451157 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:05.578444 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:10:05.578331 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:10:05.628895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.628861 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:05.700090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.700061 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:10:05.702549 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:10:05.702515 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249c2884_3f00_46d0_84a6_66bcb96f35bc.slice/crio-ab60bca8cefae7a4928646aa6bbbc21ab56582c3dda7c177c230e99b4625b0e0 WatchSource:0}: Error finding container ab60bca8cefae7a4928646aa6bbbc21ab56582c3dda7c177c230e99b4625b0e0: Status 404 returned error can't find the container with id ab60bca8cefae7a4928646aa6bbbc21ab56582c3dda7c177c230e99b4625b0e0 Nov 03 09:10:05.705584 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.705552 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:05.705675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.705616 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:05.705675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:05.705647 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:06.468069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:06.468030 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerStarted","Data":"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453"} Nov 03 09:10:06.468561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:06.468081 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerStarted","Data":"ab60bca8cefae7a4928646aa6bbbc21ab56582c3dda7c177c230e99b4625b0e0"} Nov 03 09:10:06.606216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:06.603152 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podStartSLOduration=2.6031148440000003 podStartE2EDuration="2.603114844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:06.601499331 +0000 UTC m=+1830.269540073" watchObservedRunningTime="2026-11-03 09:10:06.603114844 +0000 UTC m=+1830.271155589" Nov 03 09:10:07.250833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:07.250799 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:07.256561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:07.256537 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:07.474641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:07.474608 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:07.475601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:07.475581 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:10:09.359449 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:09.359415 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:09.574166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:09.574101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:09.846430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:09.846389 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:10.935946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:10.935915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:12.687628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:12.687591 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:12.862334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:12.862299 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:15.775396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:15.775106 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:35964->10.134.0.78:15021: read: connection reset by peer" start-of-body= Nov 03 09:10:15.775396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:15.775183 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:35964->10.134.0.78:15021: read: connection reset by peer" Nov 03 09:10:16.454259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.454230 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:16.558052 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.558023 4543 generic.go:356] "Generic (PLEG): container finished" podID="6ec19697-784d-49f5-833b-f1e146f77627" containerID="ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9" exitCode=0 Nov 03 09:10:16.558261 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.558065 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerDied","Data":"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9"} Nov 03 09:10:16.558617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.558597 4543 scope.go:122] "RemoveContainer" containerID="ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9" Nov 03 09:10:16.561115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.561072 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:16.561443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.561417 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:16.561627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:16.561609 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:10:17.568542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:17.568507 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerStarted","Data":"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254"} Nov 03 09:10:18.306389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:18.306344 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:10:18.306604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:18.306558 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 09:10:18.306683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:18.306616 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 09:10:19.305164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:19.304504 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 09:10:19.305164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:19.304581 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 09:10:20.307617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:20.307578 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 09:10:20.308871 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:20.308813 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 09:10:21.305597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:21.305555 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 09:10:21.305899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:21.305860 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 09:10:22.080443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:22.080410 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:22.306165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:22.305520 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 03 09:10:22.306165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:22.305593 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 03 09:10:23.308454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:23.308423 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:10:23.308965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:23.308931 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:10:23.309734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:23.309717 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:10:26.066206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:26.066166 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:27.247879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:27.247847 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:31.133246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:31.133214 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:32.376377 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:32.376340 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:34.778535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:34.778506 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:35.561537 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:10:35.561500 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:10:35.657379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:35.657348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:36.836798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:36.836758 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:36.976170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:36.976128 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:37.114738 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:37.114696 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:37.250356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:37.250322 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:37.948042 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:37.947989 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:39.932878 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.932840 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 09:10:39.933288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.933036 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" containerID="cri-o://f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b" gracePeriod=30 Nov 03 09:10:39.982892 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.982856 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 09:10:39.983125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.983100 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" containerID="cri-o://8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236" gracePeriod=30 Nov 03 09:10:39.985848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.985822 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 09:10:39.986034 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:39.986014 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" containerID="cri-o://7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0" gracePeriod=30 Nov 03 09:10:40.033428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:40.033393 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:40.804880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:40.804839 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:41.562381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:41.562342 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:10:41.562951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:41.562920 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:10:42.020601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:42.020560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:42.235070 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:42.235032 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:42.541456 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:42.541423 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:43.072806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:43.072774 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:10:43.073250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:43.072840 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:10:43.180312 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:43.180281 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8572v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:10:43.180515 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:43.180342 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:10:45.134934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.134900 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:45.311210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.311178 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 09:10:45.377594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.377564 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 09:10:45.386255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.386229 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 09:10:45.420059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.420032 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:45.440973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.440945 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441042 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441181 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441249 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441272 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441314 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441425 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441443 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.441513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441457 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\") pod \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\" (UID: \"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad\") " Nov 03 09:10:45.441680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.441913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441807 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.441913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.441863 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.442343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.442317 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.442443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.442348 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.442443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.442364 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.442443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.442379 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.442443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.442393 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:45.443543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.443520 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.443649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.443603 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:10:45.444043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.444027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "kube-api-access-b58nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.444382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.444367 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy" pod "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" (UID: "afb8d8a1-5124-42a2-a9ab-6d54df2d89ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.543856 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.543968 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544006 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544081 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544109 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544195 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544243 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544282 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\") pod \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\" (UID: \"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544327 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544361 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544397 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544513 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544563 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544596 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544636 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544690 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544698 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.544748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\") pod \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\" (UID: \"88989000-c596-4d8c-8eb7-4d0eb5bd86bc\") " Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545002 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545350 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545571 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545594 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545607 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545623 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545641 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545658 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545670 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.545684 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b58nt\" (UniqueName: \"kubernetes.io/projected/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad-kube-api-access-b58nt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.546071 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.546220 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.546620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.546487 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.548336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.546748 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.548336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.546814 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.548336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.547000 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:45.548336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.547360 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.549790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.549760 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "kube-api-access-g9r47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.551581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.551543 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.551581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.551557 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:10:45.551770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.551626 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:10:45.551770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.551688 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "kube-api-access-dpwf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.551900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.551877 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy" pod "88989000-c596-4d8c-8eb7-4d0eb5bd86bc" (UID: "88989000-c596-4d8c-8eb7-4d0eb5bd86bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.552236 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.552219 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:45.552333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.552319 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy" pod "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" (UID: "c7f54a4f-fc67-4f41-a7a7-64ffbefea13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:45.634108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.634077 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:45.646674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646645 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646673 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646687 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646702 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwf8\" (UniqueName: \"kubernetes.io/projected/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-kube-api-access-dpwf8\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646717 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646731 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646744 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646756 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9r47\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-kube-api-access-g9r47\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646769 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646781 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646795 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646806 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646819 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646832 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.646882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.646844 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88989000-c596-4d8c-8eb7-4d0eb5bd86bc-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:45.788269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.788231 4543 generic.go:356] "Generic (PLEG): container finished" podID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerID="f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b" exitCode=0 Nov 03 09:10:45.788490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.788357 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" event={"ID":"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad","Type":"ContainerDied","Data":"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b"} Nov 03 09:10:45.788490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.788413 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" event={"ID":"afb8d8a1-5124-42a2-a9ab-6d54df2d89ad","Type":"ContainerDied","Data":"1e49b80430f664eecdb6117ba92706a256daaa873934e5b31eeb1fcbcd391a44"} Nov 03 09:10:45.788490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.788441 4543 scope.go:122] "RemoveContainer" containerID="f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b" Nov 03 09:10:45.788667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.788608 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw" Nov 03 09:10:45.796712 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.796648 4543 generic.go:356] "Generic (PLEG): container finished" podID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerID="7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0" exitCode=0 Nov 03 09:10:45.796985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.796937 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" Nov 03 09:10:45.796985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.796977 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" event={"ID":"88989000-c596-4d8c-8eb7-4d0eb5bd86bc","Type":"ContainerDied","Data":"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0"} Nov 03 09:10:45.797115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.797027 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v" event={"ID":"88989000-c596-4d8c-8eb7-4d0eb5bd86bc","Type":"ContainerDied","Data":"d39f05aa823b44d06e14289e6a4bb614a5d80c38e5f8577e83f97c7a31037a75"} Nov 03 09:10:45.799288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.799261 4543 generic.go:356] "Generic (PLEG): container finished" podID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerID="8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236" exitCode=0 Nov 03 09:10:45.799427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.799408 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" event={"ID":"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b","Type":"ContainerDied","Data":"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236"} Nov 03 09:10:45.799476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.799457 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" event={"ID":"c7f54a4f-fc67-4f41-a7a7-64ffbefea13b","Type":"ContainerDied","Data":"54b9a994731dec30bb0e5048283e9a77a3bee40cb57ee5a9094a2a3cc49d9821"} Nov 03 09:10:45.799545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.799532 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp" Nov 03 09:10:45.811437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.811415 4543 scope.go:122] "RemoveContainer" containerID="f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b" Nov 03 09:10:45.811796 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:10:45.811774 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b\": container with ID starting with f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b not found: ID does not exist" containerID="f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b" Nov 03 09:10:45.811888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.811810 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b"} err="failed to get container status \"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b\": rpc error: code = NotFound desc = could not find container \"f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b\": container with ID starting with f435404898796bc8c2c196ea2980c3744319b15ac0c8ab4f1810606ed1aed24b not found: ID does not exist" Nov 03 09:10:45.811888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.811835 4543 scope.go:122] "RemoveContainer" containerID="7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0" Nov 03 09:10:45.832633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.832603 4543 scope.go:122] "RemoveContainer" containerID="7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0" Nov 03 09:10:45.833113 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:10:45.833081 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0\": container with ID starting with 7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0 not found: ID does not exist" containerID="7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0" Nov 03 09:10:45.833250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.833120 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0"} err="failed to get container status \"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0\": rpc error: code = NotFound desc = could not find container \"7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0\": container with ID starting with 7777a966f8ed1f8fced453aafd467dd4b1f8bcbfa33c1dbd406e588ea777acf0 not found: ID does not exist" Nov 03 09:10:45.833250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.833161 4543 scope.go:122] "RemoveContainer" containerID="8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236" Nov 03 09:10:45.843457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.843440 4543 scope.go:122] "RemoveContainer" containerID="8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236" Nov 03 09:10:45.843767 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:10:45.843748 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236\": container with ID starting with 8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236 not found: ID does not exist" containerID="8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236" Nov 03 09:10:45.843823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.843778 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236"} err="failed to get container status \"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236\": rpc error: code = NotFound desc = could not find container \"8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236\": container with ID starting with 8d00ae113d3c6c65f686652d7ba8ea993ad91064cd3c648fbd75f765a82eb236 not found: ID does not exist" Nov 03 09:10:45.960641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.960607 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 09:10:45.966345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:45.966317 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-lk9tp"] Nov 03 09:10:46.080356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.080265 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 09:10:46.085955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.085927 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8572v"] Nov 03 09:10:46.198965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.198934 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 09:10:46.204908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.204882 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-zs7bw"] Nov 03 09:10:46.453237 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.453205 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" path="/var/lib/kubelet/pods/88989000-c596-4d8c-8eb7-4d0eb5bd86bc/volumes" Nov 03 09:10:46.453766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.453744 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" path="/var/lib/kubelet/pods/afb8d8a1-5124-42a2-a9ab-6d54df2d89ad/volumes" Nov 03 09:10:46.454267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:46.454250 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" path="/var/lib/kubelet/pods/c7f54a4f-fc67-4f41-a7a7-64ffbefea13b/volumes" Nov 03 09:10:47.545865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:47.545825 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:48.617412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:48.617377 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:10:48.617932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:48.617608 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" containerID="cri-o://bcff47c8e582f166c3d287281dfd2d1a7672751c06387f03521707d212c7a78c" gracePeriod=30 Nov 03 09:10:50.563339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:50.563307 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:53.857782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.857752 4543 generic.go:356] "Generic (PLEG): container finished" podID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerID="bcff47c8e582f166c3d287281dfd2d1a7672751c06387f03521707d212c7a78c" exitCode=0 Nov 03 09:10:53.858203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.857819 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" event={"ID":"27eff15c-f1fa-48cb-92c2-80ad44d06b40","Type":"ContainerDied","Data":"bcff47c8e582f166c3d287281dfd2d1a7672751c06387f03521707d212c7a78c"} Nov 03 09:10:53.886211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.886127 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:10:53.943351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943375 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943401 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943467 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943497 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943527 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943603 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943702 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\") pod \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\" (UID: \"27eff15c-f1fa-48cb-92c2-80ad44d06b40\") " Nov 03 09:10:53.943852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943748 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.943951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.943870 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.944104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944078 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.944104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944093 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:10:53.944260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944216 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.944260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944238 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.944367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944107 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.944367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944251 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.944367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.944313 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:53.945831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.945806 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:10:53.945981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.945962 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:10:53.946053 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.946030 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:53.946280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:53.946262 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8" pod "27eff15c-f1fa-48cb-92c2-80ad44d06b40" (UID: "27eff15c-f1fa-48cb-92c2-80ad44d06b40"). InnerVolumeSpecName "kube-api-access-6f4c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:10:54.045990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.045959 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.045990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.045992 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f4c8\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-kube-api-access-6f4c8\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.046212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.046008 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.046212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.046022 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.046212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.046037 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27eff15c-f1fa-48cb-92c2-80ad44d06b40-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:10:54.865646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.865617 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" Nov 03 09:10:54.866072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.865652 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f" event={"ID":"27eff15c-f1fa-48cb-92c2-80ad44d06b40","Type":"ContainerDied","Data":"13fb53755ae663b0b0a036f88339fe90d90ff3b307b73deffbbd4a1946d05d47"} Nov 03 09:10:54.866072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:54.865719 4543 scope.go:122] "RemoveContainer" containerID="bcff47c8e582f166c3d287281dfd2d1a7672751c06387f03521707d212c7a78c" Nov 03 09:10:55.087502 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:55.087456 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:10:55.137782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:55.137712 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-gnd2f"] Nov 03 09:10:56.056494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:56.056462 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:10:56.451814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:56.451782 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" path="/var/lib/kubelet/pods/27eff15c-f1fa-48cb-92c2-80ad44d06b40/volumes" Nov 03 09:10:59.494993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:10:59.494961 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:01.020252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:01.020214 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:01.602173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:01.602126 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:05.560565 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:05.560525 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:11:06.829968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:06.829908 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:07.759411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:07.759353 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:08.974361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:08.974330 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:09.111799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:09.111762 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:09.257258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:09.257173 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:09.944425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:09.944337 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:11.451176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:11.451129 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:11.582386 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:11.582351 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:12.487857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:12.487827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:13.917342 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:13.917309 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:14.058926 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:14.058890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:14.533955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:14.533929 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lj9bz_6b6ed322-bd89-4c55-b650-28d37f9f208e/payload-processing/0.log" Nov 03 09:11:14.763041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:14.763014 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lj9bz_6b6ed322-bd89-4c55-b650-28d37f9f208e/payload-processing/0.log" Nov 03 09:11:15.173371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:15.173341 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:16.829186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:16.829152 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:17.425648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.425614 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:11:17.426192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426164 4543 cpu_manager.go:414] "containerMap: removing container" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" Nov 03 09:11:17.426192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426183 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" Nov 03 09:11:17.426192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426200 4543 cpu_manager.go:414] "containerMap: removing container" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426205 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426242 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426248 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426255 4543 cpu_manager.go:414] "containerMap: removing container" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426259 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426331 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="88989000-c596-4d8c-8eb7-4d0eb5bd86bc" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426339 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb8d8a1-5124-42a2-a9ab-6d54df2d89ad" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426346 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="27eff15c-f1fa-48cb-92c2-80ad44d06b40" containerName="istio-proxy" Nov 03 09:11:17.426453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.426356 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f54a4f-fc67-4f41-a7a7-64ffbefea13b" containerName="istio-proxy" Nov 03 09:11:17.429890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.429870 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.488428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.488395 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:11:17.534365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.534338 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-77da2a-serving-cert\"" Nov 03 09:11:17.540982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.540956 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.541127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.541105 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvcp\" (UniqueName: \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.541209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.541193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.541368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.541349 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.621749 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.621711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:17.643021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.642979 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.643021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.643023 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.643273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.643093 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvcp\" (UniqueName: \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.643273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.643184 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.643273 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:17.643245 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-serving-cert: secret "maas-api-e2e-ipp-a-77da2a-serving-cert" not found Nov 03 09:11:17.643403 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:17.643332 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls podName:86eb44dc-810b-40c3-b745-b5fe98827359 nodeName:}" failed. No retries permitted until 2026-11-03 09:11:18.143310727 +0000 UTC m=+1901.811351445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls") pod "maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" (UID: "86eb44dc-810b-40c3-b745-b5fe98827359") : secret "maas-api-e2e-ipp-a-77da2a-serving-cert" not found Nov 03 09:11:17.643823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.643798 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.645748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.645719 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.702901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.702831 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvcp\" (UniqueName: \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:17.874748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:17.874710 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:18.073803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:18.073720 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:18.149501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:18.149464 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:18.151882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:18.151864 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") pod \"maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:18.348402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:18.348315 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:18.728467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:18.728440 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:11:18.729632 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:11:18.729599 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86eb44dc_810b_40c3_b745_b5fe98827359.slice/crio-e21c43d243a62679b9061c3bd059aff5d77a85eceb0d534fb6e905aedb0558be WatchSource:0}: Error finding container e21c43d243a62679b9061c3bd059aff5d77a85eceb0d534fb6e905aedb0558be: Status 404 returned error can't find the container with id e21c43d243a62679b9061c3bd059aff5d77a85eceb0d534fb6e905aedb0558be Nov 03 09:11:19.037200 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:19.037097 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" event={"ID":"86eb44dc-810b-40c3-b745-b5fe98827359","Type":"ContainerStarted","Data":"e21c43d243a62679b9061c3bd059aff5d77a85eceb0d534fb6e905aedb0558be"} Nov 03 09:11:20.045531 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:20.045491 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" event={"ID":"86eb44dc-810b-40c3-b745-b5fe98827359","Type":"ContainerStarted","Data":"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d"} Nov 03 09:11:20.045977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:20.045731 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:20.163904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:20.163850 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" podStartSLOduration=2.299937094 podStartE2EDuration="3.163837092s" totalImagesPullingTime="863.899998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:11:20.162548039 +0000 UTC m=+1903.830588779" watchObservedRunningTime="2026-11-03 09:11:20.163837092 +0000 UTC m=+1903.831877830" Nov 03 09:11:22.169432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.169401 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:11:22.169823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.169589 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" podUID="32135f10-d3de-4a0d-896a-7b47658f9b21" containerName="maas-api" containerID="cri-o://10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968" gracePeriod=30 Nov 03 09:11:22.268328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.268293 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:22.441988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.441961 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:11:22.515797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.515756 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") pod \"32135f10-d3de-4a0d-896a-7b47658f9b21\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " Nov 03 09:11:22.515964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.515809 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\" (UniqueName: \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\") pod \"32135f10-d3de-4a0d-896a-7b47658f9b21\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " Nov 03 09:11:22.515964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.515931 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\") pod \"32135f10-d3de-4a0d-896a-7b47658f9b21\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " Nov 03 09:11:22.515964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.515958 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\") pod \"32135f10-d3de-4a0d-896a-7b47658f9b21\" (UID: \"32135f10-d3de-4a0d-896a-7b47658f9b21\") " Nov 03 09:11:22.516473 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.516438 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca" pod "32135f10-d3de-4a0d-896a-7b47658f9b21" (UID: "32135f10-d3de-4a0d-896a-7b47658f9b21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:22.516853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.516826 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32135f10-d3de-4a0d-896a-7b47658f9b21-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:22.518343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.518315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls" pod "32135f10-d3de-4a0d-896a-7b47658f9b21" (UID: "32135f10-d3de-4a0d-896a-7b47658f9b21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:11:22.518822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.518804 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt" pod "32135f10-d3de-4a0d-896a-7b47658f9b21" (UID: "32135f10-d3de-4a0d-896a-7b47658f9b21"). InnerVolumeSpecName "kube-api-access-tnnnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:22.519380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.519357 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs" pod "32135f10-d3de-4a0d-896a-7b47658f9b21" (UID: "32135f10-d3de-4a0d-896a-7b47658f9b21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:11:22.618239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.618190 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:22.618239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.618232 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32135f10-d3de-4a0d-896a-7b47658f9b21-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:22.618481 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:22.618248 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnnnt\" (UniqueName: \"kubernetes.io/projected/32135f10-d3de-4a0d-896a-7b47658f9b21-kube-api-access-tnnnt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:23.065913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.065875 4543 generic.go:356] "Generic (PLEG): container finished" podID="32135f10-d3de-4a0d-896a-7b47658f9b21" containerID="10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968" exitCode=0 Nov 03 09:11:23.065913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.065915 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" event={"ID":"32135f10-d3de-4a0d-896a-7b47658f9b21","Type":"ContainerDied","Data":"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968"} Nov 03 09:11:23.066204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.065939 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" Nov 03 09:11:23.066204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.065961 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz" event={"ID":"32135f10-d3de-4a0d-896a-7b47658f9b21","Type":"ContainerDied","Data":"24d576777a19bcadc29940727aa425202193aa62bf06d883c5c72f1da34d63b0"} Nov 03 09:11:23.066204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.065983 4543 scope.go:122] "RemoveContainer" containerID="10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968" Nov 03 09:11:23.076947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.076925 4543 scope.go:122] "RemoveContainer" containerID="10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968" Nov 03 09:11:23.077248 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:23.077230 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968\": container with ID starting with 10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968 not found: ID does not exist" containerID="10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968" Nov 03 09:11:23.077316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.077256 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968"} err="failed to get container status \"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968\": rpc error: code = NotFound desc = could not find container \"10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968\": container with ID starting with 10e822d0f731f3dbd60f8780cf6194661811e765ed17d4fafe8e5e7834f1c968 not found: ID does not exist" Nov 03 09:11:23.243500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.243465 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:11:23.294077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:23.294045 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b0c653f4-5d85856c5d-h4mpz"] Nov 03 09:11:24.102066 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:24.102030 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:24.451182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:24.451133 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32135f10-d3de-4a0d-896a-7b47658f9b21" path="/var/lib/kubelet/pods/32135f10-d3de-4a0d-896a-7b47658f9b21/volumes" Nov 03 09:11:26.057392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:26.057356 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:11:28.540418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:28.540377 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:30.606530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:30.606484 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:31.025942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:31.025883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:31.162643 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:31.162614 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:32.667332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:32.667272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:33.234827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:33.234797 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:11:33.235063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:33.235018 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" containerID="cri-o://5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9" gracePeriod=30 Nov 03 09:11:33.236382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:33.236361 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:34.089772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:34.089725 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:34.229387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:34.229319 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:34.360951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:34.360839 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:34.707704 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:34.707635 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:35.261343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:35.261308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:35.570301 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:35.570210 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:11:35.805369 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:35.805335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:36.363992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.363957 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:11:36.364576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.364561 4543 cpu_manager.go:414] "containerMap: removing container" podUID="32135f10-d3de-4a0d-896a-7b47658f9b21" containerName="maas-api" Nov 03 09:11:36.364620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.364577 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32135f10-d3de-4a0d-896a-7b47658f9b21" containerName="maas-api" Nov 03 09:11:36.364710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.364701 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="32135f10-d3de-4a0d-896a-7b47658f9b21" containerName="maas-api" Nov 03 09:11:36.369576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.369554 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.396060 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.396010 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:36.424068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.424043 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ae3db\"" Nov 03 09:11:36.489165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.489109 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ae3db-dockercfg-rbvvk\"" Nov 03 09:11:36.489372 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.489182 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:11:36.524083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.524024 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz49w\" (UniqueName: \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.524496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.524470 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.626442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.626351 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz49w\" (UniqueName: \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.626601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.626475 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.627162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.627126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.688739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.688714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz49w\" (UniqueName: \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:36.802314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.802274 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:36.940100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.940013 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:36.983222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:36.983190 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:37.363797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:37.363769 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:11:37.364498 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:11:37.364470 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac22672_e40d_4ab4_bc2f_5f5c078af90f.slice/crio-c0cc9fc5af83c039413fa75e650e8dd17f934e1866bb4256d7fad7e7b6051212 WatchSource:0}: Error finding container c0cc9fc5af83c039413fa75e650e8dd17f934e1866bb4256d7fad7e7b6051212: Status 404 returned error can't find the container with id c0cc9fc5af83c039413fa75e650e8dd17f934e1866bb4256d7fad7e7b6051212 Nov 03 09:11:38.164299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.164261 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" event={"ID":"4ac22672-e40d-4ab4-bc2f-5f5c078af90f","Type":"ContainerStarted","Data":"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e"} Nov 03 09:11:38.164485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.164324 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" event={"ID":"4ac22672-e40d-4ab4-bc2f-5f5c078af90f","Type":"ContainerStarted","Data":"c0cc9fc5af83c039413fa75e650e8dd17f934e1866bb4256d7fad7e7b6051212"} Nov 03 09:11:38.164485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.164390 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:38.285496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.285446 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" podStartSLOduration=2.285428385 podStartE2EDuration="2.285428385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:11:38.283208353 +0000 UTC m=+1921.951249093" watchObservedRunningTime="2026-11-03 09:11:38.285428385 +0000 UTC m=+1921.953469124" Nov 03 09:11:38.307453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.307415 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:38.531020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.530991 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:11:38.551335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551304 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551341 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551366 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551408 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551445 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551488 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551524 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.551766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.551555 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.552249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552076 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:38.552249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552209 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:38.552249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552218 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:38.552249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552204 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:38.552554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552263 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:11:38.552554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.552362 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\") pod \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\" (UID: \"0e86e01d-e600-4cab-b543-6c39d5f1dac6\") " Nov 03 09:11:38.554319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.554274 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.554440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.554321 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.554440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.554337 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.554440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.554351 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.554440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.554365 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.563689 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.563653 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:11:38.563829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.563715 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:38.563829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.563735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "kube-api-access-g86p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:11:38.563829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.563760 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy" pod "0e86e01d-e600-4cab-b543-6c39d5f1dac6" (UID: "0e86e01d-e600-4cab-b543-6c39d5f1dac6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:11:38.655327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.655298 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.655327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.655324 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.655532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.655336 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g86p8\" (UniqueName: \"kubernetes.io/projected/0e86e01d-e600-4cab-b543-6c39d5f1dac6-kube-api-access-g86p8\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:38.655532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:38.655351 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e86e01d-e600-4cab-b543-6c39d5f1dac6-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:11:39.172154 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.172103 4543 generic.go:356] "Generic (PLEG): container finished" podID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerID="5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9" exitCode=0 Nov 03 09:11:39.172349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.172171 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerDied","Data":"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9"} Nov 03 09:11:39.172349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.172212 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" Nov 03 09:11:39.172349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.172263 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z" event={"ID":"0e86e01d-e600-4cab-b543-6c39d5f1dac6","Type":"ContainerDied","Data":"db0ba4b1c8c554d87dc5241a62c4444945061413512143d09005dd8275c3f98e"} Nov 03 09:11:39.172349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.172289 4543 scope.go:122] "RemoveContainer" containerID="5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9" Nov 03 09:11:39.182679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.182661 4543 scope.go:122] "RemoveContainer" containerID="0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2" Nov 03 09:11:39.192303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.192282 4543 scope.go:122] "RemoveContainer" containerID="5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9" Nov 03 09:11:39.192623 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:39.192603 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9\": container with ID starting with 5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9 not found: ID does not exist" containerID="5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9" Nov 03 09:11:39.192680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.192637 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9"} err="failed to get container status \"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9\": rpc error: code = NotFound desc = could not find container \"5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9\": container with ID starting with 5bfc3411ded24cecf691e92984e0cc167fadb3d60d766948a9528f0c528c38d9 not found: ID does not exist" Nov 03 09:11:39.192680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.192667 4543 scope.go:122] "RemoveContainer" containerID="0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2" Nov 03 09:11:39.192946 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:39.192926 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2\": container with ID starting with 0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2 not found: ID does not exist" containerID="0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2" Nov 03 09:11:39.193010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.192952 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2"} err="failed to get container status \"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2\": rpc error: code = NotFound desc = could not find container \"0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2\": container with ID starting with 0ff218b9dd1d24ad8590a42d1568f9a06ba1bdcec756ee643470382fed479fd2 not found: ID does not exist" Nov 03 09:11:39.393314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.393265 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:11:39.444281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.444196 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-840559-openshift-default-c66b5d46c-26b4z"] Nov 03 09:11:39.714104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.714018 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:39.851606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:39.851545 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:40.380491 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:40.380462 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:40.451941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:40.451908 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" path="/var/lib/kubelet/pods/0e86e01d-e600-4cab-b543-6c39d5f1dac6/volumes" Nov 03 09:11:41.721441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:41.721404 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:44.055843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:44.055815 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:47.732997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:47.732914 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:47.962716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:47.962684 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:48.106226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:48.105711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:49.175033 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:49.175005 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:11:50.811836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.811799 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:11:50.812357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812339 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.812530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812510 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.812594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812542 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.812594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812548 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.812707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812697 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.812739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.812709 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e86e01d-e600-4cab-b543-6c39d5f1dac6" containerName="istio-proxy" Nov 03 09:11:50.817678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.817662 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:50.875978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.875949 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c78f2b-serving-cert\"" Nov 03 09:11:50.879637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.879604 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:11:50.921560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.921531 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:50.921762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.921631 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:50.921868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.921844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:50.921948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:50.921931 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h8bl\" (UniqueName: \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.013392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.013353 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:51.023853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.023823 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.023987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.023904 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h8bl\" (UniqueName: \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.024348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.024032 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.024348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.024102 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.024348 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:51.024129 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-serving-cert: secret "maas-api-e2e-ipp-b-c78f2b-serving-cert" not found Nov 03 09:11:51.024348 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:11:51.024216 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls podName:0fe5019e-f9ea-4c45-9031-54210d78181f nodeName:}" failed. No retries permitted until 2026-11-03 09:11:51.524195129 +0000 UTC m=+1935.192235865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls") pod "maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" (UID: "0fe5019e-f9ea-4c45-9031-54210d78181f") : secret "maas-api-e2e-ipp-b-c78f2b-serving-cert" not found Nov 03 09:11:51.024807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.024787 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.026782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.026764 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.112569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.112485 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h8bl\" (UniqueName: \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.167110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.167076 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:51.304181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.304131 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:51.532113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.532036 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.535123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.535099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:51.732900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:51.732869 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:52.135297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:52.135269 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:11:52.135970 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:11:52.135948 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe5019e_f9ea_4c45_9031_54210d78181f.slice/crio-6c493fac6f0fcf1bd942e60fd5cfe66ba1a93c895eb96f132b5705835ba575f0 WatchSource:0}: Error finding container 6c493fac6f0fcf1bd942e60fd5cfe66ba1a93c895eb96f132b5705835ba575f0: Status 404 returned error can't find the container with id 6c493fac6f0fcf1bd942e60fd5cfe66ba1a93c895eb96f132b5705835ba575f0 Nov 03 09:11:52.264591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:52.264548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" event={"ID":"0fe5019e-f9ea-4c45-9031-54210d78181f","Type":"ContainerStarted","Data":"6c493fac6f0fcf1bd942e60fd5cfe66ba1a93c895eb96f132b5705835ba575f0"} Nov 03 09:11:53.273206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:53.273131 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" event={"ID":"0fe5019e-f9ea-4c45-9031-54210d78181f","Type":"ContainerStarted","Data":"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c"} Nov 03 09:11:53.273630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:53.273238 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:11:53.397845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:53.397792 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" podStartSLOduration=2.424528306 podStartE2EDuration="3.397778451s" totalImagesPullingTime="973.250145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:11:53.39634687 +0000 UTC m=+1937.064387609" watchObservedRunningTime="2026-11-03 09:11:53.397778451 +0000 UTC m=+1937.065819189" Nov 03 09:11:56.258818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:56.258778 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:11:57.299374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.299346 4543 generic.go:356] "Generic (PLEG): container finished" podID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerID="b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453" exitCode=0 Nov 03 09:11:57.299772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.299420 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerDied","Data":"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453"} Nov 03 09:11:57.299817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.299804 4543 scope.go:122] "RemoveContainer" containerID="b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453" Nov 03 09:11:57.301277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.301248 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:11:57.301384 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.301330 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:11:57.301447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:57.301380 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:11:58.312655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:58.312598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerStarted","Data":"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41"} Nov 03 09:11:59.250778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:59.250746 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:11:59.251040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:59.251018 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 09:11:59.251090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:59.251068 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 09:11:59.307422 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:11:59.307393 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:12:00.250707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:00.250675 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 09:12:00.251258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:00.250739 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 09:12:00.646389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:00.646342 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:01.250580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:01.250542 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 09:12:01.250798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:01.250612 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 09:12:01.970649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:01.970618 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 09:12:01.970864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:01.970840 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" containerID="cri-o://d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224" gracePeriod=30 Nov 03 09:12:02.250458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:02.250384 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 09:12:02.250458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:02.250431 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 09:12:02.432364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:02.432329 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:03.250803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:03.250773 4543 patch_prober.go:29] interesting pod/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 09:12:03.250994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:03.250836 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 09:12:03.633207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:03.633170 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:03.770775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:03.770739 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:04.254940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:04.254912 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:12:04.255414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:04.255391 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:12:04.256175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:04.256130 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:12:05.249193 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:05.249156 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:05.560789 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:05.560703 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:12:05.791899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:05.791867 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:05.796439 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:05.796415 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:12:05.796656 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:05.796627 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" containerID="cri-o://3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254" gracePeriod=30 Nov 03 09:12:07.239252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.239192 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 09:12:07.328381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328346 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328426 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328496 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328539 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328579 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328656 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:07.328735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328710 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328765 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328803 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.328887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.328862 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\") pod \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\" (UID: \"1a93cd66-8b94-4ba0-bbc3-448d61c19de3\") " Nov 03 09:12:07.329054 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329023 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:07.329577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329319 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:07.329719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329574 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:07.329884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329740 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.329884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329759 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.329884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329827 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.329884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329843 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.329884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.329855 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:07.331541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.331500 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:07.331662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.331545 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "kube-api-access-gf2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:07.331821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.331803 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:07.332519 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.332495 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy" pod "1a93cd66-8b94-4ba0-bbc3-448d61c19de3" (UID: "1a93cd66-8b94-4ba0-bbc3-448d61c19de3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:07.370123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.370096 4543 generic.go:356] "Generic (PLEG): container finished" podID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerID="d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224" exitCode=0 Nov 03 09:12:07.370252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.370166 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" Nov 03 09:12:07.370252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.370168 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerDied","Data":"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224"} Nov 03 09:12:07.370252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.370230 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj" event={"ID":"1a93cd66-8b94-4ba0-bbc3-448d61c19de3","Type":"ContainerDied","Data":"23d1e1950808eb504aa2c9287398859dcea93728fbd68f87c17e7dbc433dbaad"} Nov 03 09:12:07.370380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.370261 4543 scope.go:122] "RemoveContainer" containerID="d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224" Nov 03 09:12:07.380889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.380874 4543 scope.go:122] "RemoveContainer" containerID="310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1" Nov 03 09:12:07.392248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.392123 4543 scope.go:122] "RemoveContainer" containerID="d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224" Nov 03 09:12:07.392609 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:07.392587 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224\": container with ID starting with d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224 not found: ID does not exist" containerID="d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224" Nov 03 09:12:07.392710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.392615 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224"} err="failed to get container status \"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224\": rpc error: code = NotFound desc = could not find container \"d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224\": container with ID starting with d46af4c9fe60a303fbc3f664d1cdfdd4353f10ebb60dfd1c36b715b369e83224 not found: ID does not exist" Nov 03 09:12:07.392710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.392640 4543 scope.go:122] "RemoveContainer" containerID="310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1" Nov 03 09:12:07.392960 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:07.392927 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1\": container with ID starting with 310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1 not found: ID does not exist" containerID="310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1" Nov 03 09:12:07.393004 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.392985 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1"} err="failed to get container status \"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1\": rpc error: code = NotFound desc = could not find container \"310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1\": container with ID starting with 310a4d9b62db3193dd0975891010c5ff8c184233af5e97253133264e2f806ad1 not found: ID does not exist" Nov 03 09:12:07.431172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.431120 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.431172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.431172 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2gl\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-kube-api-access-gf2gl\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.431340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.431189 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.431340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.431205 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.431340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.431218 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a93cd66-8b94-4ba0-bbc3-448d61c19de3-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:07.525582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.525502 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 09:12:07.531314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.531284 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-p76dj"] Nov 03 09:12:07.742950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:07.742888 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:08.068128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.068097 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:08.311880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.311827 4543 patch_prober.go:29] interesting pod/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:12:08.312501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.311897 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:12:08.400847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.385536 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:08.452066 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.452032 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" path="/var/lib/kubelet/pods/1a93cd66-8b94-4ba0-bbc3-448d61c19de3/volumes" Nov 03 09:12:08.528798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:08.528769 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:09.241778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:09.241749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:11.057376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.057322 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:12:11.185295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185267 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185339 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185374 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185413 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185438 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185472 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185562 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:11.185682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:11.185877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185764 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.185877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185783 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:12:11.185877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.185850 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\") pod \"6ec19697-784d-49f5-833b-f1e146f77627\" (UID: \"6ec19697-784d-49f5-833b-f1e146f77627\") " Nov 03 09:12:11.186059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:11.186197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186175 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:11.186440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186385 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.186440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186416 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.186440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186429 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.186608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186441 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec19697-784d-49f5-833b-f1e146f77627-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.186608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.186456 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.187970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.187945 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "kube-api-access-7z7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:11.188084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.188029 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:12:11.188439 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.188410 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:12:11.188528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.188473 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy" pod "6ec19697-784d-49f5-833b-f1e146f77627" (UID: "6ec19697-784d-49f5-833b-f1e146f77627"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:12:11.287772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.287738 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7lt\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-kube-api-access-7z7lt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.287772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.287764 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec19697-784d-49f5-833b-f1e146f77627-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.287772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.287774 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec19697-784d-49f5-833b-f1e146f77627-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.287994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.287784 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec19697-784d-49f5-833b-f1e146f77627-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:12:11.427808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.427776 4543 generic.go:356] "Generic (PLEG): container finished" podID="6ec19697-784d-49f5-833b-f1e146f77627" containerID="3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254" exitCode=0 Nov 03 09:12:11.428000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.427837 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" Nov 03 09:12:11.428000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.427866 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerDied","Data":"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254"} Nov 03 09:12:11.428000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.427929 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm" event={"ID":"6ec19697-784d-49f5-833b-f1e146f77627","Type":"ContainerDied","Data":"e789e4da5d878554087149f61938d2eb07597cc50d80599feee2ce8fda5a9bf4"} Nov 03 09:12:11.428190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.428003 4543 scope.go:122] "RemoveContainer" containerID="3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254" Nov 03 09:12:11.439195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.439179 4543 scope.go:122] "RemoveContainer" containerID="ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9" Nov 03 09:12:11.448819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.448800 4543 scope.go:122] "RemoveContainer" containerID="3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254" Nov 03 09:12:11.449069 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:11.449047 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254\": container with ID starting with 3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254 not found: ID does not exist" containerID="3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254" Nov 03 09:12:11.449118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.449078 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254"} err="failed to get container status \"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254\": rpc error: code = NotFound desc = could not find container \"3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254\": container with ID starting with 3597773cb46d3d3d2af5910a50f768a8e66cce6702c6a1cf28fd344c8b224254 not found: ID does not exist" Nov 03 09:12:11.449118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.449094 4543 scope.go:122] "RemoveContainer" containerID="ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9" Nov 03 09:12:11.449348 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:11.449329 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9\": container with ID starting with ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9 not found: ID does not exist" containerID="ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9" Nov 03 09:12:11.449395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.449355 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9"} err="failed to get container status \"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9\": rpc error: code = NotFound desc = could not find container \"ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9\": container with ID starting with ad593f2b7c176a4e076e843cdda1cb84b8a3351dfe07b520ff90ebfa2ecb06f9 not found: ID does not exist" Nov 03 09:12:11.598396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.598364 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:12:11.650216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:11.650190 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bcc5ce-openshift-default-75548fc589-c58vm"] Nov 03 09:12:12.048242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.048204 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:12.253917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.253883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:12.431271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.431236 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:12.451835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.451800 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec19697-784d-49f5-833b-f1e146f77627" path="/var/lib/kubelet/pods/6ec19697-784d-49f5-833b-f1e146f77627/volumes" Nov 03 09:12:12.588797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.588759 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:12.726953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.726871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:12.897576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:12.897544 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:15.592792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:15.592761 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:15.763380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:15.763349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:15.900526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:15.900492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:16.738192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:16.738157 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:21.122675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:21.122642 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:26.163822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:26.163787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:31.227735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:31.227701 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:32.578085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:32.578047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:33.329465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:33.329374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:33.464862 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:33.464767 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:33.601684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:33.601594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:34.022563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:34.022527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:34.170922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:34.170880 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:35.561338 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:12:35.561303 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:12:36.849580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:36.849537 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:38.585642 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:38.585607 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:38.761947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:38.761912 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:38.960485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:38.960449 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:40.747122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:40.747090 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:41.501895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:41.501858 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:41.838090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:41.838012 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:42.015274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:42.015238 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:44.169722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:44.169651 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:49.361404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:49.361367 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:12:56.139250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:12:56.139218 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:01.244166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:01.244120 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:02.901935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:02.901898 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:04.167816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:04.167769 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:05.561736 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:05.561691 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:13:05.737677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:05.737644 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:06.576268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:06.576231 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:07.996324 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:07.996292 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:08.134569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:08.134523 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:11.943408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:11.943370 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:12.192375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:12.192340 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:18.942993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:18.942742 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:19.075934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:19.075904 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:19.200548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:19.200467 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:24.027932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:24.027891 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:28.971830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:28.971798 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:29.883945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:29.883911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:32.636713 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:32.636679 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:13:32.637171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:32.636889 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" containerID="cri-o://528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6" gracePeriod=30 Nov 03 09:13:34.155148 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:34.155113 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:35.559914 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:35.559876 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:13:37.878740 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.878713 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:37.896288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.896266 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:13:37.933934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.933867 4543 generic.go:356] "Generic (PLEG): container finished" podID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerID="528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6" exitCode=0 Nov 03 09:13:37.933934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.933912 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" Nov 03 09:13:37.934095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.933952 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" event={"ID":"785907e1-e6ca-4037-ab6e-d2e9ab4ee320","Type":"ContainerDied","Data":"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6"} Nov 03 09:13:37.934095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.934001 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt" event={"ID":"785907e1-e6ca-4037-ab6e-d2e9ab4ee320","Type":"ContainerDied","Data":"452a04b6a946c13eac80e75c7c594df21926987d4b1822b5c8e4118b7f49e578"} Nov 03 09:13:37.934095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.934049 4543 scope.go:122] "RemoveContainer" containerID="528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6" Nov 03 09:13:37.943365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.943348 4543 scope.go:122] "RemoveContainer" containerID="528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6" Nov 03 09:13:37.943607 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:37.943591 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6\": container with ID starting with 528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6 not found: ID does not exist" containerID="528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6" Nov 03 09:13:37.943649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.943619 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6"} err="failed to get container status \"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6\": rpc error: code = NotFound desc = could not find container \"528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6\": container with ID starting with 528129b184845e5d9949f5f640091a665692a3c477ffadadfb57352b52f7e8d6 not found: ID does not exist" Nov 03 09:13:37.993322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993284 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993354 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993386 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993429 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993461 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993503 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993538 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993643 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.993767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.993748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\") pod \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\" (UID: \"785907e1-e6ca-4037-ab6e-d2e9ab4ee320\") " Nov 03 09:13:37.995380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.995069 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:13:37.996334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.995965 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:13:37.996334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.996038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:13:37.996334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.996091 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:13:37.996922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:37.996886 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:38.008982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.007431 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:38.008982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.008039 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:13:38.014874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.014738 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:13:38.019159 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.017379 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7" pod "785907e1-e6ca-4037-ab6e-d2e9ab4ee320" (UID: "785907e1-e6ca-4037-ab6e-d2e9ab4ee320"). InnerVolumeSpecName "kube-api-access-wzwr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:38.096499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096463 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzwr7\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-kube-api-access-wzwr7\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096494 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096504 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096513 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096521 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096529 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096537 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096546 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.096806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.096558 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785907e1-e6ca-4037-ab6e-d2e9ab4ee320-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:38.360101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.360070 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:13:38.365404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.365379 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-cszxt"] Nov 03 09:13:38.398864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.398836 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:38.450567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:38.450541 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" path="/var/lib/kubelet/pods/785907e1-e6ca-4037-ab6e-d2e9ab4ee320/volumes" Nov 03 09:13:40.475928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:40.475896 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:40.613082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:40.613048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:41.145347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:41.145312 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:41.924214 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:41.924179 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:45.879523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.879492 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:13:45.880025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880012 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880025 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880038 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:13:45.880068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880043 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:13:45.880068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880057 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:13:45.880068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880063 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880078 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880082 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880100 4543 cpu_manager.go:414] "containerMap: removing container" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880104 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880190 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880197 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a93cd66-8b94-4ba0-bbc3-448d61c19de3" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880204 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:13:45.880255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.880212 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="785907e1-e6ca-4037-ab6e-d2e9ab4ee320" containerName="istio-proxy" Nov 03 09:13:45.884814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.884798 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:45.942069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.942045 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:13:45.982261 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.982227 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knnjt\" (UniqueName: \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:45.982409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.982388 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:45.982446 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.982419 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:45.982497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.982490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:45.988658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:45.988631 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-121ee6-serving-cert\"" Nov 03 09:13:46.071102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.071073 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:46.083322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.083300 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.083417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.083361 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knnjt\" (UniqueName: \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.083462 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:46.083443 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-serving-cert: secret "maas-api-e2e-ipp-b-121ee6-serving-cert" not found Nov 03 09:13:46.083515 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:46.083503 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls podName:58344624-e496-48e4-8815-c5da35945f06 nodeName:}" failed. No retries permitted until 2026-11-03 09:13:46.583486998 +0000 UTC m=+2050.251527734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls") pod "maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" (UID: "58344624-e496-48e4-8815-c5da35945f06") : secret "maas-api-e2e-ipp-b-121ee6-serving-cert" not found Nov 03 09:13:46.083568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.083522 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.083568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.083539 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.084221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.084203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.086113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.086093 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.143320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.143222 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knnjt\" (UniqueName: \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.231682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.231634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:46.487184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.487072 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:46.591025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.590992 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.593513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.593490 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") pod \"maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.797744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.797667 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:46.824318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:46.824179 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lj9bz_6b6ed322-bd89-4c55-b650-28d37f9f208e/payload-processing/0.log" Nov 03 09:13:47.055978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:47.055908 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lj9bz_6b6ed322-bd89-4c55-b650-28d37f9f208e/payload-processing/0.log" Nov 03 09:13:47.175540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:47.175511 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:13:47.176373 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:13:47.176349 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58344624_e496_48e4_8815_c5da35945f06.slice/crio-fbba1c36cc374bebc15f1d1722c978d598087dd95bf94f7307461cb826679a39 WatchSource:0}: Error finding container fbba1c36cc374bebc15f1d1722c978d598087dd95bf94f7307461cb826679a39: Status 404 returned error can't find the container with id fbba1c36cc374bebc15f1d1722c978d598087dd95bf94f7307461cb826679a39 Nov 03 09:13:47.993672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:47.993636 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" event={"ID":"58344624-e496-48e4-8815-c5da35945f06","Type":"ContainerStarted","Data":"fbba1c36cc374bebc15f1d1722c978d598087dd95bf94f7307461cb826679a39"} Nov 03 09:13:49.000647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:49.000599 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" event={"ID":"58344624-e496-48e4-8815-c5da35945f06","Type":"ContainerStarted","Data":"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c"} Nov 03 09:13:49.001129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:49.000739 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:49.118356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:49.118296 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" podStartSLOduration=3.313347865 podStartE2EDuration="4.118276413s" totalImagesPullingTime="804.928548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:13:49.117124523 +0000 UTC m=+2052.785165261" watchObservedRunningTime="2026-11-03 09:13:49.118276413 +0000 UTC m=+2052.786317153" Nov 03 09:13:50.645373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:50.645345 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:51.181170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.181110 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:13:51.181409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.181381 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" podUID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" containerName="payload-pre-processing" containerID="cri-o://3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e" gracePeriod=30 Nov 03 09:13:51.200040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.200018 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:51.329980 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.329949 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:51.436183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.436092 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:13:51.545329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.545295 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\") pod \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " Nov 03 09:13:51.545493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.545469 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\" (UniqueName: \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\") pod \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\" (UID: \"4ac22672-e40d-4ab4-bc2f-5f5c078af90f\") " Nov 03 09:13:51.545715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.545695 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume" pod "4ac22672-e40d-4ab4-bc2f-5f5c078af90f" (UID: "4ac22672-e40d-4ab4-bc2f-5f5c078af90f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:13:51.546336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.546314 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:51.547662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.547641 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w" pod "4ac22672-e40d-4ab4-bc2f-5f5c078af90f" (UID: "4ac22672-e40d-4ab4-bc2f-5f5c078af90f"). InnerVolumeSpecName "kube-api-access-zz49w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:13:51.647915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:51.647877 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz49w\" (UniqueName: \"kubernetes.io/projected/4ac22672-e40d-4ab4-bc2f-5f5c078af90f-kube-api-access-zz49w\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:13:52.021570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.021535 4543 generic.go:356] "Generic (PLEG): container finished" podID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" containerID="3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e" exitCode=0 Nov 03 09:13:52.021741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.021583 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" Nov 03 09:13:52.021741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.021621 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" event={"ID":"4ac22672-e40d-4ab4-bc2f-5f5c078af90f","Type":"ContainerDied","Data":"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e"} Nov 03 09:13:52.021741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.021676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp" event={"ID":"4ac22672-e40d-4ab4-bc2f-5f5c078af90f","Type":"ContainerDied","Data":"c0cc9fc5af83c039413fa75e650e8dd17f934e1866bb4256d7fad7e7b6051212"} Nov 03 09:13:52.021741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.021698 4543 scope.go:122] "RemoveContainer" containerID="3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e" Nov 03 09:13:52.031897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.031880 4543 scope.go:122] "RemoveContainer" containerID="3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e" Nov 03 09:13:52.032179 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:13:52.032151 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e\": container with ID starting with 3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e not found: ID does not exist" containerID="3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e" Nov 03 09:13:52.032286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.032185 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e"} err="failed to get container status \"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e\": rpc error: code = NotFound desc = could not find container \"3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e\": container with ID starting with 3259c65cd766f392899763e94aed6df80b73fb8458cdc0ba1e25262cbd19007e not found: ID does not exist" Nov 03 09:13:52.192866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.192800 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:13:52.196244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.196218 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-59885f8c6c-jdcrp"] Nov 03 09:13:52.453257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.453226 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" path="/var/lib/kubelet/pods/4ac22672-e40d-4ab4-bc2f-5f5c078af90f/volumes" Nov 03 09:13:52.521195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:52.521167 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:55.015309 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:55.015284 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:13:56.845510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:56.845473 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:56.983950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:56.983916 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:57.729437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:57.729403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:58.493667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:58.493628 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:13:59.017803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:13:59.017771 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:00.706331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:00.706293 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:03.250152 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:03.250107 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:05.117210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:05.117159 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:05.561015 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:14:05.560930 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:14:05.768757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:05.768721 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:08.867834 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:08.867795 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:09.387274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:09.387241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:11.865525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:11.865496 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:12.152560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:12.152482 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:17.181863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:17.181826 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:18.833256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:18.833225 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:22.822132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:22.822095 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:29.551708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:29.551647 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:30.204378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:30.204346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:34.653696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:34.653663 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:35.560809 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:14:35.560772 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:14:36.693323 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:36.693298 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:14:36.700028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:36.700009 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:14:40.264906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:40.264872 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:40.396847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:40.396808 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:42.271007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:42.270959 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:42.411086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:42.411046 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:44.128866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:44.128830 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:45.939927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:45.939889 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:46.076681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:46.076646 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:50.371043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:50.370959 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:51.392169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:51.392082 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:52.372246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:52.372216 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:52.713882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:52.713788 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:52.966533 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:52.966428 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:56.750633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:56.750598 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:56.890062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:56.890036 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:57.129539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:57.129508 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:57.277943 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:57.277908 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:14:58.423652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:14:58.423608 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:00.217544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:00.217510 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:00.399119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:00.399088 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:02.667563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.667525 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:15:02.668680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.668654 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" containerName="payload-pre-processing" Nov 03 09:15:02.668680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.668677 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" containerName="payload-pre-processing" Nov 03 09:15:02.668867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.668802 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec19697-784d-49f5-833b-f1e146f77627" containerName="istio-proxy" Nov 03 09:15:02.668867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.668812 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac22672-e40d-4ab4-bc2f-5f5c078af90f" containerName="payload-pre-processing" Nov 03 09:15:02.672253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.672231 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.781579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.781537 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:15:02.823435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823398 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823558 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823581 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5j5\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823645 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823695 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823765 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823718 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823806 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.823857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.823847 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925041 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925081 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925187 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925239 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925283 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925383 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5j5\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925461 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925680 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925722 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925738 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.925864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.925814 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.926188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.926132 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.927764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.927744 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.927852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.927790 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.985063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.985033 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.985387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.985364 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5j5\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:02.986037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:02.986021 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:03.417298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.417266 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:15:03.417700 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:15:03.417673 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e03bd68_c33c_44a3_9e77_7c4c32d7f179.slice/crio-47cf31462a545ab261979d98a082f62cd8fd96e1db76ce3e3981444efec1d04e WatchSource:0}: Error finding container 47cf31462a545ab261979d98a082f62cd8fd96e1db76ce3e3981444efec1d04e: Status 404 returned error can't find the container with id 47cf31462a545ab261979d98a082f62cd8fd96e1db76ce3e3981444efec1d04e Nov 03 09:15:03.420535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.420506 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:03.420615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.420576 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:03.420615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.420605 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:03.444964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.444934 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" event={"ID":"9e03bd68-c33c-44a3-9e77-7c4c32d7f179","Type":"ContainerStarted","Data":"47cf31462a545ab261979d98a082f62cd8fd96e1db76ce3e3981444efec1d04e"} Nov 03 09:15:03.638793 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:03.638737 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:04.135987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:04.135951 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:04.452700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:04.452601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" event={"ID":"9e03bd68-c33c-44a3-9e77-7c4c32d7f179","Type":"ContainerStarted","Data":"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c"} Nov 03 09:15:04.626606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:04.626551 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podStartSLOduration=2.626538005 podStartE2EDuration="2.626538005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:04.624529392 +0000 UTC m=+2128.292570130" watchObservedRunningTime="2026-11-03 09:15:04.626538005 +0000 UTC m=+2128.294578742" Nov 03 09:15:04.994591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:04.989485 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:05.002465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:05.002433 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:05.002714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:05.002683 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:05.561455 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:05.561423 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:15:05.995284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:05.995250 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:05.995913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:05.995534 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:07.002271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.002230 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:07.002821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.002791 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:07.155841 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.155766 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:07.479328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.479265 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:07.621310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.621282 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:08.001166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.996506 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:08.001166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:07.996580 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:08.711124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:08.711084 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:15:08.711607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:08.711376 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" containerID="cri-o://e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a" gracePeriod=30 Nov 03 09:15:08.809408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:08.805886 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:08.989028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:08.988564 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:08.989028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:08.988626 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:09.025594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.025567 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:15:09.115714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.115685 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\") pod \"457aa574-2c41-41be-bbda-59384f60203b\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " Nov 03 09:15:09.115919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.115787 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\") pod \"457aa574-2c41-41be-bbda-59384f60203b\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " Nov 03 09:15:09.115919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.115881 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\" (UniqueName: \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\") pod \"457aa574-2c41-41be-bbda-59384f60203b\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " Nov 03 09:15:09.116064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.115979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") pod \"457aa574-2c41-41be-bbda-59384f60203b\" (UID: \"457aa574-2c41-41be-bbda-59384f60203b\") " Nov 03 09:15:09.116064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.116047 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca" pod "457aa574-2c41-41be-bbda-59384f60203b" (UID: "457aa574-2c41-41be-bbda-59384f60203b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:09.116590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.116563 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/457aa574-2c41-41be-bbda-59384f60203b-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.118393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.118365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5" pod "457aa574-2c41-41be-bbda-59384f60203b" (UID: "457aa574-2c41-41be-bbda-59384f60203b"). InnerVolumeSpecName "kube-api-access-kq7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:09.118585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.118560 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls" pod "457aa574-2c41-41be-bbda-59384f60203b" (UID: "457aa574-2c41-41be-bbda-59384f60203b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:09.118643 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.118570 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs" pod "457aa574-2c41-41be-bbda-59384f60203b" (UID: "457aa574-2c41-41be-bbda-59384f60203b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:09.125709 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.125629 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:09.217818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.217778 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7q5\" (UniqueName: \"kubernetes.io/projected/457aa574-2c41-41be-bbda-59384f60203b-kube-api-access-kq7q5\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.217818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.217823 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.218014 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.217841 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/457aa574-2c41-41be-bbda-59384f60203b-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.275718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.275645 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:15:09.275962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.275913 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" podUID="d4f97bab-2199-43e5-acca-1181cfb09ce9" containerName="payload-pre-processing" containerID="cri-o://5e03022165e291142cfd45610c65b99aacd08a9155377d001aa15a2f6b6bf1a8" gracePeriod=30 Nov 03 09:15:09.366309 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.366279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:09.485449 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.485418 4543 generic.go:356] "Generic (PLEG): container finished" podID="457aa574-2c41-41be-bbda-59384f60203b" containerID="e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a" exitCode=0 Nov 03 09:15:09.485627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.485473 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" Nov 03 09:15:09.485627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.485509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerDied","Data":"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a"} Nov 03 09:15:09.485627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.485574 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s" event={"ID":"457aa574-2c41-41be-bbda-59384f60203b","Type":"ContainerDied","Data":"37d8eab536658176914ebe2565016bcd8bc0f3359cde15510283c5f444c55fa7"} Nov 03 09:15:09.485627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.485603 4543 scope.go:122] "RemoveContainer" containerID="e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a" Nov 03 09:15:09.490536 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.489829 4543 generic.go:356] "Generic (PLEG): container finished" podID="d4f97bab-2199-43e5-acca-1181cfb09ce9" containerID="5e03022165e291142cfd45610c65b99aacd08a9155377d001aa15a2f6b6bf1a8" exitCode=0 Nov 03 09:15:09.490536 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.490040 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" event={"ID":"d4f97bab-2199-43e5-acca-1181cfb09ce9","Type":"ContainerDied","Data":"5e03022165e291142cfd45610c65b99aacd08a9155377d001aa15a2f6b6bf1a8"} Nov 03 09:15:09.501757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.501728 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:09.504830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.504804 4543 scope.go:122] "RemoveContainer" containerID="52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5" Nov 03 09:15:09.515062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.515038 4543 scope.go:122] "RemoveContainer" containerID="e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a" Nov 03 09:15:09.515381 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:09.515348 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a\": container with ID starting with e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a not found: ID does not exist" containerID="e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a" Nov 03 09:15:09.515487 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.515391 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a"} err="failed to get container status \"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a\": rpc error: code = NotFound desc = could not find container \"e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a\": container with ID starting with e76c111d2ad376c92d852157a22b8bdd389fe589782bb8b66813c6b57fd2e46a not found: ID does not exist" Nov 03 09:15:09.515487 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.515414 4543 scope.go:122] "RemoveContainer" containerID="52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5" Nov 03 09:15:09.515689 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:09.515672 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5\": container with ID starting with 52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5 not found: ID does not exist" containerID="52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5" Nov 03 09:15:09.515741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.515698 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5"} err="failed to get container status \"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5\": rpc error: code = NotFound desc = could not find container \"52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5\": container with ID starting with 52db8a1a39f3f44da091476009f6b13480862da04b4f31d31e8cb2864ebe2fe5 not found: ID does not exist" Nov 03 09:15:09.538325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.538304 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:15:09.627044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.627004 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\") pod \"d4f97bab-2199-43e5-acca-1181cfb09ce9\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " Nov 03 09:15:09.627236 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.627125 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\" (UniqueName: \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\") pod \"d4f97bab-2199-43e5-acca-1181cfb09ce9\" (UID: \"d4f97bab-2199-43e5-acca-1181cfb09ce9\") " Nov 03 09:15:09.627441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.627415 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume" pod "d4f97bab-2199-43e5-acca-1181cfb09ce9" (UID: "d4f97bab-2199-43e5-acca-1181cfb09ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:09.627826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.627761 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4f97bab-2199-43e5-acca-1181cfb09ce9-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.629304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.629283 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9" pod "d4f97bab-2199-43e5-acca-1181cfb09ce9" (UID: "d4f97bab-2199-43e5-acca-1181cfb09ce9"). InnerVolumeSpecName "kube-api-access-vbtg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:09.656292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.656249 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:15:09.659420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.659394 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04934e-5c687df6bd-7ld5s"] Nov 03 09:15:09.728834 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.728806 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbtg9\" (UniqueName: \"kubernetes.io/projected/d4f97bab-2199-43e5-acca-1181cfb09ce9-kube-api-access-vbtg9\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:09.986454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.986420 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 09:15:09.986627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:09.986473 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 09:15:10.088777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.088740 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:15:10.088993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.088968 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" podUID="0fe5019e-f9ea-4c45-9031-54210d78181f" containerName="maas-api" containerID="cri-o://9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c" gracePeriod=30 Nov 03 09:15:10.175625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.175594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:10.361406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.361374 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:15:10.467392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.467352 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457aa574-2c41-41be-bbda-59384f60203b" path="/var/lib/kubelet/pods/457aa574-2c41-41be-bbda-59384f60203b/volumes" Nov 03 09:15:10.503744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.503706 4543 generic.go:356] "Generic (PLEG): container finished" podID="0fe5019e-f9ea-4c45-9031-54210d78181f" containerID="9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c" exitCode=0 Nov 03 09:15:10.504041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.504013 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" event={"ID":"0fe5019e-f9ea-4c45-9031-54210d78181f","Type":"ContainerDied","Data":"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c"} Nov 03 09:15:10.504267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.504247 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" event={"ID":"0fe5019e-f9ea-4c45-9031-54210d78181f","Type":"ContainerDied","Data":"6c493fac6f0fcf1bd942e60fd5cfe66ba1a93c895eb96f132b5705835ba575f0"} Nov 03 09:15:10.504417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.504401 4543 scope.go:122] "RemoveContainer" containerID="9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c" Nov 03 09:15:10.504698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.504676 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w" Nov 03 09:15:10.509232 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.509210 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" event={"ID":"d4f97bab-2199-43e5-acca-1181cfb09ce9","Type":"ContainerDied","Data":"b5a3c47a43d778b7317bc4368be1a012aabc13b8e9bae4c94223a66c1485dbc9"} Nov 03 09:15:10.511335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.511315 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv" Nov 03 09:15:10.528151 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.528104 4543 scope.go:122] "RemoveContainer" containerID="9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c" Nov 03 09:15:10.529040 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:10.529004 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c\": container with ID starting with 9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c not found: ID does not exist" containerID="9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c" Nov 03 09:15:10.529176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.529049 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c"} err="failed to get container status \"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c\": rpc error: code = NotFound desc = could not find container \"9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c\": container with ID starting with 9d4f23fb23a19b5f045beb59a17176a6dbd2bee2a340963ac1553bcf0ed36c1c not found: ID does not exist" Nov 03 09:15:10.529176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.529074 4543 scope.go:122] "RemoveContainer" containerID="5e03022165e291142cfd45610c65b99aacd08a9155377d001aa15a2f6b6bf1a8" Nov 03 09:15:10.544673 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.544622 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\") pod \"0fe5019e-f9ea-4c45-9031-54210d78181f\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " Nov 03 09:15:10.544782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.544735 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\" (UniqueName: \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\") pod \"0fe5019e-f9ea-4c45-9031-54210d78181f\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " Nov 03 09:15:10.544846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.544800 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") pod \"0fe5019e-f9ea-4c45-9031-54210d78181f\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " Nov 03 09:15:10.544846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.544833 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\") pod \"0fe5019e-f9ea-4c45-9031-54210d78181f\" (UID: \"0fe5019e-f9ea-4c45-9031-54210d78181f\") " Nov 03 09:15:10.546560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.546115 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca" pod "0fe5019e-f9ea-4c45-9031-54210d78181f" (UID: "0fe5019e-f9ea-4c45-9031-54210d78181f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:10.550501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.550475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs" pod "0fe5019e-f9ea-4c45-9031-54210d78181f" (UID: "0fe5019e-f9ea-4c45-9031-54210d78181f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:10.550653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.550633 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls" pod "0fe5019e-f9ea-4c45-9031-54210d78181f" (UID: "0fe5019e-f9ea-4c45-9031-54210d78181f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:10.552591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.552565 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl" pod "0fe5019e-f9ea-4c45-9031-54210d78181f" (UID: "0fe5019e-f9ea-4c45-9031-54210d78181f"). InnerVolumeSpecName "kube-api-access-8h8bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:10.647666 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.647633 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.647837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.647671 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe5019e-f9ea-4c45-9031-54210d78181f-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.647837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.647688 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe5019e-f9ea-4c45-9031-54210d78181f-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.647837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.647703 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h8bl\" (UniqueName: \"kubernetes.io/projected/0fe5019e-f9ea-4c45-9031-54210d78181f-kube-api-access-8h8bl\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:10.684274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.684209 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:15:10.689636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.689596 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-6c6b58746c-46fwv"] Nov 03 09:15:10.796363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.796274 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:10.925490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.925456 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:10.932086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.932055 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:15:10.943250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.943207 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c78f2b-db8547bc6-4q48w"] Nov 03 09:15:10.990941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.990912 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:10.991325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.991303 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:10.992047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:10.992028 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:15:11.388361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:11.388331 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:11.517783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:11.517752 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:12.131650 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:12.131621 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:12.454091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:12.454004 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe5019e-f9ea-4c45-9031-54210d78181f" path="/var/lib/kubelet/pods/0fe5019e-f9ea-4c45-9031-54210d78181f/volumes" Nov 03 09:15:12.454499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:12.454479 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f97bab-2199-43e5-acca-1181cfb09ce9" path="/var/lib/kubelet/pods/d4f97bab-2199-43e5-acca-1181cfb09ce9/volumes" Nov 03 09:15:15.222889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:15.222855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:15.352391 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:15.352358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:16.754971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:16.754940 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:16.940102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:16.940069 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:17.331027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.330997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:17.468278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.468230 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:17.661458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661416 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:15:17.661953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661936 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0fe5019e-f9ea-4c45-9031-54210d78181f" containerName="maas-api" Nov 03 09:15:17.661953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661950 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe5019e-f9ea-4c45-9031-54210d78181f" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661962 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d4f97bab-2199-43e5-acca-1181cfb09ce9" containerName="payload-pre-processing" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661967 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f97bab-2199-43e5-acca-1181cfb09ce9" containerName="payload-pre-processing" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661980 4543 cpu_manager.go:414] "containerMap: removing container" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.661985 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.662008 4543 cpu_manager.go:414] "containerMap: removing container" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.662012 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.662082 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.662094 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe5019e-f9ea-4c45-9031-54210d78181f" containerName="maas-api" Nov 03 09:15:17.662118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.662100 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f97bab-2199-43e5-acca-1181cfb09ce9" containerName="payload-pre-processing" Nov 03 09:15:17.666804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.666783 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.724448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.724413 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:15:17.757485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757454 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.757847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.757847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757508 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.757847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq92f\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.757847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757707 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.758007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757847 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.758007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757893 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.758007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.757961 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.758007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.758005 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.858969 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859034 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859060 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859088 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859229 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq92f\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859275 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859346 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859377 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859412 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859453 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859496 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859747 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.859958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.859935 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.861791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.861768 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.861975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.861960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.918463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.918404 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.918728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.918707 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq92f\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:17.982650 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:17.982618 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:18.102932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.102896 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:18.234215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.234113 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:18.415467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.415438 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:15:18.416436 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:15:18.416407 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e730438_f249_4bec_b0d5_10a63c7cb36b.slice/crio-0c4c3beaa1c665569d98a6e9ca143fb7e5b4292f09b70842b4ec4c94e17fedde WatchSource:0}: Error finding container 0c4c3beaa1c665569d98a6e9ca143fb7e5b4292f09b70842b4ec4c94e17fedde: Status 404 returned error can't find the container with id 0c4c3beaa1c665569d98a6e9ca143fb7e5b4292f09b70842b4ec4c94e17fedde Nov 03 09:15:18.419428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.419396 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:18.419530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.419479 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:18.419530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.419521 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:15:18.559708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.559608 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" event={"ID":"8e730438-f249-4bec-b0d5-10a63c7cb36b","Type":"ContainerStarted","Data":"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631"} Nov 03 09:15:18.559708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.559676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" event={"ID":"8e730438-f249-4bec-b0d5-10a63c7cb36b","Type":"ContainerStarted","Data":"0c4c3beaa1c665569d98a6e9ca143fb7e5b4292f09b70842b4ec4c94e17fedde"} Nov 03 09:15:18.691447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.691397 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podStartSLOduration=1.691383947 podStartE2EDuration="1.691383947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:18.689916735 +0000 UTC m=+2142.357957499" watchObservedRunningTime="2026-11-03 09:15:18.691383947 +0000 UTC m=+2142.359424684" Nov 03 09:15:18.983038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:18.983007 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:20.012213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:20.006610 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:20.012213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:20.006677 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:20.987204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:20.985515 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:20.987204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:20.985573 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:21.999620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:21.999585 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:22.000242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:22.000205 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:22.845859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:22.845825 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:23.000165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:22.999252 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:23.000165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:22.999313 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:23.045729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:23.045694 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:24.006291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:24.006248 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:24.006792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:24.006316 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:24.987166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:24.983332 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 09:15:24.987166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:24.983403 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 09:15:27.004639 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.004590 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:27.024256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.024228 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:15:27.024442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.024286 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:15:27.143397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.143348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:27.355370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.355291 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:27.355591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.355571 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:27.356419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:27.356396 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:15:29.476962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:29.476928 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:29.931415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:29.931387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:32.306561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:32.306518 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:32.769251 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:32.769218 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:35.426743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:35.426712 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:35.561911 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:35.561872 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:15:40.530241 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:40.530202 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:42.610180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:42.610117 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:44.808536 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:44.808502 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:15:44.808951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:44.808703 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" podUID="86eb44dc-810b-40c3-b745-b5fe98827359" containerName="maas-api" containerID="cri-o://2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d" gracePeriod=30 Nov 03 09:15:44.898732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:44.898656 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:45.088183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.088157 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:15:45.180963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.180933 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") pod \"86eb44dc-810b-40c3-b745-b5fe98827359\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " Nov 03 09:15:45.181121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.180988 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\") pod \"86eb44dc-810b-40c3-b745-b5fe98827359\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " Nov 03 09:15:45.181121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.181055 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\") pod \"86eb44dc-810b-40c3-b745-b5fe98827359\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " Nov 03 09:15:45.181266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.181198 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\" (UniqueName: \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\") pod \"86eb44dc-810b-40c3-b745-b5fe98827359\" (UID: \"86eb44dc-810b-40c3-b745-b5fe98827359\") " Nov 03 09:15:45.181742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.181399 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca" pod "86eb44dc-810b-40c3-b745-b5fe98827359" (UID: "86eb44dc-810b-40c3-b745-b5fe98827359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:15:45.181742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.181706 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86eb44dc-810b-40c3-b745-b5fe98827359-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.183736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.183695 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls" pod "86eb44dc-810b-40c3-b745-b5fe98827359" (UID: "86eb44dc-810b-40c3-b745-b5fe98827359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:45.193227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.193180 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp" pod "86eb44dc-810b-40c3-b745-b5fe98827359" (UID: "86eb44dc-810b-40c3-b745-b5fe98827359"). InnerVolumeSpecName "kube-api-access-9hvcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:15:45.195508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.195469 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs" pod "86eb44dc-810b-40c3-b745-b5fe98827359" (UID: "86eb44dc-810b-40c3-b745-b5fe98827359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:15:45.282516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.282483 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.282516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.282536 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hvcp\" (UniqueName: \"kubernetes.io/projected/86eb44dc-810b-40c3-b745-b5fe98827359-kube-api-access-9hvcp\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.282516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.282550 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86eb44dc-810b-40c3-b745-b5fe98827359-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:15:45.438929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.438897 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:45.569441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.569403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:45.763332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.763256 4543 generic.go:356] "Generic (PLEG): container finished" podID="86eb44dc-810b-40c3-b745-b5fe98827359" containerID="2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d" exitCode=0 Nov 03 09:15:45.763574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.763293 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" event={"ID":"86eb44dc-810b-40c3-b745-b5fe98827359","Type":"ContainerDied","Data":"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d"} Nov 03 09:15:45.763719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.763614 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" event={"ID":"86eb44dc-810b-40c3-b745-b5fe98827359","Type":"ContainerDied","Data":"e21c43d243a62679b9061c3bd059aff5d77a85eceb0d534fb6e905aedb0558be"} Nov 03 09:15:45.763719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.763327 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v" Nov 03 09:15:45.763719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.763649 4543 scope.go:122] "RemoveContainer" containerID="2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d" Nov 03 09:15:45.773535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.773515 4543 scope.go:122] "RemoveContainer" containerID="2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d" Nov 03 09:15:45.773847 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:45.773827 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d\": container with ID starting with 2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d not found: ID does not exist" containerID="2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d" Nov 03 09:15:45.773924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.773857 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d"} err="failed to get container status \"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d\": rpc error: code = NotFound desc = could not find container \"2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d\": container with ID starting with 2c40dc0b7d16c440bd9ddc801bfe0b570f5df47c24dbc4a34752576985ed573d not found: ID does not exist" Nov 03 09:15:45.958774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.958744 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:15:45.961198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:45.961176 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-77da2a-6696858f99-x9v9v"] Nov 03 09:15:46.430476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:46.430447 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:46.450594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:46.450565 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86eb44dc-810b-40c3-b745-b5fe98827359" path="/var/lib/kubelet/pods/86eb44dc-810b-40c3-b745-b5fe98827359/volumes" Nov 03 09:15:46.794185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:46.794098 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:51.724178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.724130 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:15:51.724717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.724703 4543 cpu_manager.go:414] "containerMap: removing container" podUID="86eb44dc-810b-40c3-b745-b5fe98827359" containerName="maas-api" Nov 03 09:15:51.724779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.724719 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86eb44dc-810b-40c3-b745-b5fe98827359" containerName="maas-api" Nov 03 09:15:51.724830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.724821 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="86eb44dc-810b-40c3-b745-b5fe98827359" containerName="maas-api" Nov 03 09:15:51.724872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.724834 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="457aa574-2c41-41be-bbda-59384f60203b" containerName="maas-api" Nov 03 09:15:51.729728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.729708 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.749160 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.749123 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.749272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.749257 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.749315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.749303 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs55x\" (UniqueName: \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.749425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.749407 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.834383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.834346 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-0eecfd-serving-cert\"" Nov 03 09:15:51.837120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.837099 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:15:51.850696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.850669 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.850824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.850731 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs55x\" (UniqueName: \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.850824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.850815 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.850938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.850923 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.851075 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:51.851054 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-serving-cert: secret "maas-api-e2e-praxis-b-0eecfd-serving-cert" not found Nov 03 09:15:51.851154 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:15:51.851130 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls podName:cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa nodeName:}" failed. No retries permitted until 2026-11-03 09:15:52.351109815 +0000 UTC m=+2176.019150547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls") pod "maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" (UID: "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa") : secret "maas-api-e2e-praxis-b-0eecfd-serving-cert" not found Nov 03 09:15:51.851464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.851401 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.853175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.853117 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.919482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.919452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs55x\" (UniqueName: \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:51.938369 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:51.938336 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:52.084237 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:52.084123 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:52.285310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:52.285274 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:52.357071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:52.356999 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:52.359497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:52.359474 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:52.643049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:52.643015 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:53.025655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:53.025627 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:15:53.026761 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:15:53.026730 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2f6e49_50aa_48ae_b01d_e7b1c06d19fa.slice/crio-9e90591fe376334d4256d2535077e65f2b8d2803543692bf7c212b78b6c71cd2 WatchSource:0}: Error finding container 9e90591fe376334d4256d2535077e65f2b8d2803543692bf7c212b78b6c71cd2: Status 404 returned error can't find the container with id 9e90591fe376334d4256d2535077e65f2b8d2803543692bf7c212b78b6c71cd2 Nov 03 09:15:53.028133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:53.028114 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:15:53.809655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:53.809624 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:53.811520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:53.811492 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerStarted","Data":"9e90591fe376334d4256d2535077e65f2b8d2803543692bf7c212b78b6c71cd2"} Nov 03 09:15:54.015459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:54.015432 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:54.820875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:54.820839 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerStarted","Data":"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646"} Nov 03 09:15:54.821431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:54.820951 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:15:54.939520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:54.939460 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" podStartSLOduration=3.019396436 podStartE2EDuration="3.939445604s" totalImagesPullingTime="920.049168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:54.938746229 +0000 UTC m=+2178.606786959" watchObservedRunningTime="2026-11-03 09:15:54.939445604 +0000 UTC m=+2178.607486342" Nov 03 09:15:55.769531 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:55.769491 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:56.263353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:56.263316 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:15:56.263846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:56.263605 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" containerID="cri-o://16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41" gracePeriod=30 Nov 03 09:15:56.290315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:56.290290 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:57.070806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:57.070776 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:15:58.460918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:15:58.460886 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:00.844128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:00.844089 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.89:8443/health\": dial tcp 10.134.0.89:8443: connect: connection refused" Nov 03 09:16:01.525364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.525342 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:16:01.602583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602512 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602620 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602649 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602690 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602858 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602791 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602858 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602817 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602859 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.602955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602945 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:01.603126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602965 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:01.603126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.602978 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:01.603126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603056 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.603126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603090 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\") pod \"249c2884-3f00-46d0-84a6-66bcb96f35bc\" (UID: \"249c2884-3f00-46d0-84a6-66bcb96f35bc\") " Nov 03 09:16:01.603400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603220 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:01.603400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603333 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:01.603946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603919 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.603946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603947 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.604161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603962 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.604161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603976 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249c2884-3f00-46d0-84a6-66bcb96f35bc-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.604161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.603992 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.605020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.604986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:01.605205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.605184 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:01.605273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.605255 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "kube-api-access-bv27z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:01.605345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.605327 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo" pod "249c2884-3f00-46d0-84a6-66bcb96f35bc" (UID: "249c2884-3f00-46d0-84a6-66bcb96f35bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:16:01.704888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.704853 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv27z\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-kube-api-access-bv27z\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.704888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.704889 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.705094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.704900 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.705094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.704908 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249c2884-3f00-46d0-84a6-66bcb96f35bc-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:01.877505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.877470 4543 generic.go:356] "Generic (PLEG): container finished" podID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerID="16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41" exitCode=0 Nov 03 09:16:01.877973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.877518 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" Nov 03 09:16:01.877973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.877549 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerDied","Data":"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41"} Nov 03 09:16:01.877973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.877619 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6" event={"ID":"249c2884-3f00-46d0-84a6-66bcb96f35bc","Type":"ContainerDied","Data":"ab60bca8cefae7a4928646aa6bbbc21ab56582c3dda7c177c230e99b4625b0e0"} Nov 03 09:16:01.877973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.877652 4543 scope.go:122] "RemoveContainer" containerID="16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41" Nov 03 09:16:01.888247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.888227 4543 scope.go:122] "RemoveContainer" containerID="b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453" Nov 03 09:16:01.896875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.896855 4543 scope.go:122] "RemoveContainer" containerID="16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41" Nov 03 09:16:01.897168 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:01.897124 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41\": container with ID starting with 16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41 not found: ID does not exist" containerID="16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41" Nov 03 09:16:01.897221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.897178 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41"} err="failed to get container status \"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41\": rpc error: code = NotFound desc = could not find container \"16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41\": container with ID starting with 16b1c0dbe023fbd8404cdf540a0731bbf22be1ffa7206414af7fb49e27747e41 not found: ID does not exist" Nov 03 09:16:01.897221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.897200 4543 scope.go:122] "RemoveContainer" containerID="b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453" Nov 03 09:16:01.897447 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:01.897429 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453\": container with ID starting with b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453 not found: ID does not exist" containerID="b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453" Nov 03 09:16:01.897497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:01.897451 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453"} err="failed to get container status \"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453\": rpc error: code = NotFound desc = could not find container \"b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453\": container with ID starting with b7407c67d75d29c61d4b4c9d0f3ceba81dee19d6b898f2db5ec98e83304c8453 not found: ID does not exist" Nov 03 09:16:02.050112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:02.050083 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:16:02.056874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:02.056852 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-77da2a-openshift-default-566c4df7c5-mn5j6"] Nov 03 09:16:02.451524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:02.451495 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" path="/var/lib/kubelet/pods/249c2884-3f00-46d0-84a6-66bcb96f35bc/volumes" Nov 03 09:16:02.848607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:02.848528 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:03.327424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:03.327384 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:04.041853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:04.041811 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:04.691952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:04.691907 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:04.904664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:04.904624 4543 generic.go:356] "Generic (PLEG): container finished" podID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerID="e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646" exitCode=1 Nov 03 09:16:04.904821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:04.904702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerDied","Data":"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646"} Nov 03 09:16:04.905190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:04.905173 4543 scope.go:122] "RemoveContainer" containerID="e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646" Nov 03 09:16:05.560061 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:05.560028 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:16:05.912875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:05.912840 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerStarted","Data":"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b"} Nov 03 09:16:05.913413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:05.913037 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:16:06.132078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:06.132048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:08.482099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.482067 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:16:08.482818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.482796 4543 cpu_manager.go:414] "containerMap: removing container" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.482916 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.482820 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.483003 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.482990 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.486419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.486399 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.532739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.532707 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:16:08.533310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.533291 4543 cpu_manager.go:414] "containerMap: removing container" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.533310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.533310 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.533468 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.533443 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="249c2884-3f00-46d0-84a6-66bcb96f35bc" containerName="istio-proxy" Nov 03 09:16:08.536655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.536637 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.580275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.580250 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-77207e-openshift-default-dockercfg-rcrfh\"" Nov 03 09:16:08.582926 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.582901 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:16:08.595453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595425 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595466 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595578 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595623 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595674 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2t8g\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595717 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.595746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595740 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595765 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595958 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.595997 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596102 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.596288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596156 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596197 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.596288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596267 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:16:08.596484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596381 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkbw\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596435 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.596484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596465 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596487 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.596640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.596513 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697518 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2t8g\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697560 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697577 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697595 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697619 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697635 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697666 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697686 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697716 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.697747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697743 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkbw\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697769 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697798 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697838 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697939 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697962 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.697995 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698033 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698079 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698094 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698130 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.698990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698676 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.698990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.698700 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.699641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.699120 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.700497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.700472 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.700665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.700645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.701090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.701068 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.701318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.701297 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.761007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.760937 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.761007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.760960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.761209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.761130 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2t8g\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.761276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.761255 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkbw\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:08.799345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.799313 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:08.853072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:08.850728 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:09.229231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.229203 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:16:09.233029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.232998 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.233120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.233063 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.233120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.233093 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.282447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.282421 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:16:09.282812 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:16:09.282792 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ceb1e6_0f29_417f_af2d_f462e90e609b.slice/crio-e5d57d45d0a30c0aea5b8203c7362f60cc6469fec00c7905e2ed9dab92d5c447 WatchSource:0}: Error finding container e5d57d45d0a30c0aea5b8203c7362f60cc6469fec00c7905e2ed9dab92d5c447: Status 404 returned error can't find the container with id e5d57d45d0a30c0aea5b8203c7362f60cc6469fec00c7905e2ed9dab92d5c447 Nov 03 09:16:09.285711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.285680 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.285782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.285744 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.285782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.285773 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:09.946876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.946838 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" event={"ID":"0d520045-3d44-4d89-bf73-571f2fdd05da","Type":"ContainerStarted","Data":"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea"} Nov 03 09:16:09.947361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.946897 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" event={"ID":"0d520045-3d44-4d89-bf73-571f2fdd05da","Type":"ContainerStarted","Data":"91ec9716c05f83467658b3b7ed1b964499a5e0a7f6b574438e796a594bb3fe87"} Nov 03 09:16:09.948531 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.948507 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" event={"ID":"89ceb1e6-0f29-417f-af2d-f462e90e609b","Type":"ContainerStarted","Data":"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195"} Nov 03 09:16:09.948626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:09.948548 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" event={"ID":"89ceb1e6-0f29-417f-af2d-f462e90e609b","Type":"ContainerStarted","Data":"e5d57d45d0a30c0aea5b8203c7362f60cc6469fec00c7905e2ed9dab92d5c447"} Nov 03 09:16:10.068031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.067982 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podStartSLOduration=2.067967027 podStartE2EDuration="2.067967027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:10.067399071 +0000 UTC m=+2193.735439811" watchObservedRunningTime="2026-11-03 09:16:10.067967027 +0000 UTC m=+2193.736007765" Nov 03 09:16:10.801467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.801434 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:10.803994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.803962 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:10.804262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.804222 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:10.851786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.851754 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:10.853185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.853132 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:10.853311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:10.853204 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:11.236722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.236678 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:11.806163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.805570 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:11.806163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.805630 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:11.851966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.851928 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:11.852164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.851984 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:11.936541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:11.936508 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:16:12.114116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:12.114048 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podStartSLOduration=4.11402694 podStartE2EDuration="4.11402694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:10.215846861 +0000 UTC m=+2193.883887600" watchObservedRunningTime="2026-11-03 09:16:12.11402694 +0000 UTC m=+2195.782067683" Nov 03 09:16:12.800029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:12.799994 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:12.800687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:12.800655 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:12.851681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:12.851647 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:12.851843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:12.851713 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:13.810164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:13.809546 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:13.810164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:13.809607 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:13.851511 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:13.851472 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:13.851684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:13.851527 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:14.804164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:14.803401 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:14.804164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:14.803489 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:14.851108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:14.851068 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:14.851497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:14.851124 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:15.799865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:15.799834 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 09:16:15.800026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:15.799897 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 09:16:15.851798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:15.851749 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:16:15.852204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:15.851826 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:16:16.236008 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.235971 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:16.804103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.804075 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:16.804542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.804528 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:16.805306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.805288 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:16:16.855121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.855059 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:16.855473 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.855331 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:16.855912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:16.855893 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:16:20.688550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:20.688519 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:16:20.688956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:20.688718 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" podUID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" containerName="maas-api" containerID="cri-o://0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2" gracePeriod=30 Nov 03 09:16:20.785627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:20.785594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:20.962043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:20.962021 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:16:21.021575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.021544 4543 generic.go:356] "Generic (PLEG): container finished" podID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" containerID="0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2" exitCode=0 Nov 03 09:16:21.021727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.021592 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" Nov 03 09:16:21.021727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.021628 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" event={"ID":"6b5c36fb-82d4-4a22-ab23-2dced0f945e8","Type":"ContainerDied","Data":"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2"} Nov 03 09:16:21.021727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.021681 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp" event={"ID":"6b5c36fb-82d4-4a22-ab23-2dced0f945e8","Type":"ContainerDied","Data":"9820c8e4043ab06effa5739ad37eb5b446ac63ada9b48ca521f3cbcd8504ae8b"} Nov 03 09:16:21.021727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.021710 4543 scope.go:122] "RemoveContainer" containerID="0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2" Nov 03 09:16:21.029174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029122 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\") pod \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " Nov 03 09:16:21.029297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029181 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\" (UniqueName: \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\") pod \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " Nov 03 09:16:21.029297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029287 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\") pod \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " Nov 03 09:16:21.029402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029321 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\") pod \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\" (UID: \"6b5c36fb-82d4-4a22-ab23-2dced0f945e8\") " Nov 03 09:16:21.029724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029698 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca" pod "6b5c36fb-82d4-4a22-ab23-2dced0f945e8" (UID: "6b5c36fb-82d4-4a22-ab23-2dced0f945e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:21.029992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.029913 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.031512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.031491 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs" pod "6b5c36fb-82d4-4a22-ab23-2dced0f945e8" (UID: "6b5c36fb-82d4-4a22-ab23-2dced0f945e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:21.031609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.031511 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4" pod "6b5c36fb-82d4-4a22-ab23-2dced0f945e8" (UID: "6b5c36fb-82d4-4a22-ab23-2dced0f945e8"). InnerVolumeSpecName "kube-api-access-696z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:21.031907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.031878 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls" pod "6b5c36fb-82d4-4a22-ab23-2dced0f945e8" (UID: "6b5c36fb-82d4-4a22-ab23-2dced0f945e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:21.032098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.032084 4543 scope.go:122] "RemoveContainer" containerID="0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2" Nov 03 09:16:21.032404 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:21.032380 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2\": container with ID starting with 0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2 not found: ID does not exist" containerID="0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2" Nov 03 09:16:21.032500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.032411 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2"} err="failed to get container status \"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2\": rpc error: code = NotFound desc = could not find container \"0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2\": container with ID starting with 0b7ce947e76ab00f2a35d07819f0387de0e742844611f1f6bd518703e8b449c2 not found: ID does not exist" Nov 03 09:16:21.131124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.131087 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.131297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.131129 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696z4\" (UniqueName: \"kubernetes.io/projected/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-kube-api-access-696z4\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.131297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.131163 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b5c36fb-82d4-4a22-ab23-2dced0f945e8-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:21.364117 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.364029 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:21.448821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.448790 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:16:21.460590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.460559 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c32d41-78d5f6944c-989vp"] Nov 03 09:16:21.502484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:21.502434 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:22.469859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:22.468456 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" path="/var/lib/kubelet/pods/6b5c36fb-82d4-4a22-ab23-2dced0f945e8/volumes" Nov 03 09:16:23.117171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.116701 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:23.649905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.649872 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:16:23.650423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.650408 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" containerName="maas-api" Nov 03 09:16:23.650477 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.650424 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" containerName="maas-api" Nov 03 09:16:23.650516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.650505 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5c36fb-82d4-4a22-ab23-2dced0f945e8" containerName="maas-api" Nov 03 09:16:23.655268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.655252 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.718974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.718935 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.718990 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719168 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719210 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719250 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719363 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fh98\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719515 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.719597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.719580 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.765561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.765537 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:16:23.820742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820713 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820770 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820819 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820842 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820871 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820886 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.820936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.820975 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821012 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fh98\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821220 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821278 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821364 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821476 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.821939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.821916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.823308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.823286 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.823537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.823516 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.880764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.880736 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.881238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.881220 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fh98\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:23.969776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:23.969710 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:24.399065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:24.399041 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:16:24.400197 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:16:24.400169 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c9017c_31c8_47c8_8d59_b6a6cd9d8835.slice/crio-d92f3a05dc6f71ed17dc45a160d88bef994a066524cc816a7bb5c83e7c1788a3 WatchSource:0}: Error finding container d92f3a05dc6f71ed17dc45a160d88bef994a066524cc816a7bb5c83e7c1788a3: Status 404 returned error can't find the container with id d92f3a05dc6f71ed17dc45a160d88bef994a066524cc816a7bb5c83e7c1788a3 Nov 03 09:16:24.402707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:24.402672 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:24.402788 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:24.402736 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:24.402788 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:24.402766 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:25.115677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:25.115635 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" event={"ID":"63c9017c-31c8-47c8-8d59-b6a6cd9d8835","Type":"ContainerStarted","Data":"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce"} Nov 03 09:16:25.116074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:25.115694 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" event={"ID":"63c9017c-31c8-47c8-8d59-b6a6cd9d8835","Type":"ContainerStarted","Data":"d92f3a05dc6f71ed17dc45a160d88bef994a066524cc816a7bb5c83e7c1788a3"} Nov 03 09:16:25.970846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:25.970814 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:25.972279 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:25.972258 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:25.972348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:25.972305 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:26.211781 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:26.211742 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:26.347197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:26.347026 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:26.970602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:26.970566 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:26.970799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:26.970625 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:27.848356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:27.848323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:27.970194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:27.970133 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:27.970350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:27.970221 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:28.345887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:28.345854 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:28.636308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:28.636254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:28.970816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:28.970739 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:28.970816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:28.970796 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:29.747636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:29.747605 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:29.970227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:29.970191 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:29.970398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:29.970256 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:30.385227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:30.377915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:30.648815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:30.648735 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:30.785831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:30.785799 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:30.970341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:30.970249 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 09:16:30.970491 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:30.970329 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 09:16:31.974724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:31.974697 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:31.975209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:31.975072 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:31.975649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:31.975630 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:16:32.097999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:32.097952 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podStartSLOduration=9.097938634 podStartE2EDuration="9.097938634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:25.336009922 +0000 UTC m=+2209.004050672" watchObservedRunningTime="2026-11-03 09:16:32.097938634 +0000 UTC m=+2215.765979447" Nov 03 09:16:34.112869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:34.112835 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:34.925463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:34.925433 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:35.109303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:35.109269 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:35.245018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:35.244941 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:35.562003 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:35.561921 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:16:37.694038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:37.694000 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:16:37.694503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:37.694222 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" containerID="cri-o://ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae" gracePeriod=30 Nov 03 09:16:37.958959 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:37.958903 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:16:38.015902 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.015874 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.015950 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.015986 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.016073 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.016095 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.016163 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.016301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.016196 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\") pod \"53853238-48e6-4f0e-96ed-5b033097689e\" (UID: \"53853238-48e6-4f0e-96ed-5b033097689e\") " Nov 03 09:16:38.017009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.016959 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:38.018409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.018384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:38.018647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.018626 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "kube-api-access-k6hk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:38.018735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.018719 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:38.019006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.018986 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:38.019288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.019269 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:16:38.019343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.019318 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts" pod "53853238-48e6-4f0e-96ed-5b033097689e" (UID: "53853238-48e6-4f0e-96ed-5b033097689e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:38.117722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117687 4543 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-kubeconfig\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117713 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117723 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hk6\" (UniqueName: \"kubernetes.io/projected/53853238-48e6-4f0e-96ed-5b033097689e-kube-api-access-k6hk6\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117733 4543 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/53853238-48e6-4f0e-96ed-5b033097689e-local-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117742 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-ca-configmap\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117751 4543 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-cacerts\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.117946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.117759 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/53853238-48e6-4f0e-96ed-5b033097689e-istio-csr-dns-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:38.189808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.189751 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:38.202358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.202333 4543 generic.go:356] "Generic (PLEG): container finished" podID="53853238-48e6-4f0e-96ed-5b033097689e" containerID="ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae" exitCode=0 Nov 03 09:16:38.202497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.202376 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" Nov 03 09:16:38.202497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.202396 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" event={"ID":"53853238-48e6-4f0e-96ed-5b033097689e","Type":"ContainerDied","Data":"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae"} Nov 03 09:16:38.202497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.202435 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk" event={"ID":"53853238-48e6-4f0e-96ed-5b033097689e","Type":"ContainerDied","Data":"ac4998f7e5553e9f8011560407af194d7685a6bd02e134b41dcb13a7e6130461"} Nov 03 09:16:38.202497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.202455 4543 scope.go:122] "RemoveContainer" containerID="ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae" Nov 03 09:16:38.212616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.212596 4543 scope.go:122] "RemoveContainer" containerID="ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae" Nov 03 09:16:38.212882 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:38.212863 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae\": container with ID starting with ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae not found: ID does not exist" containerID="ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae" Nov 03 09:16:38.212928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.212889 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae"} err="failed to get container status \"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae\": rpc error: code = NotFound desc = could not find container \"ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae\": container with ID starting with ee182ac6e37c6db4fdc9336f5600f728a25c765bf4eb275c47b2adee68f99aae not found: ID does not exist" Nov 03 09:16:38.374784 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.374739 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:38.377437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.377405 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:16:38.382550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.382522 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ldmbk"] Nov 03 09:16:38.451660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.451625 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53853238-48e6-4f0e-96ed-5b033097689e" path="/var/lib/kubelet/pods/53853238-48e6-4f0e-96ed-5b033097689e/volumes" Nov 03 09:16:38.754639 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.754604 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:16:38.755364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.755345 4543 cpu_manager.go:414] "containerMap: removing container" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" Nov 03 09:16:38.755474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.755365 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" Nov 03 09:16:38.755532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.755523 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="53853238-48e6-4f0e-96ed-5b033097689e" containerName="discovery" Nov 03 09:16:38.760349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.760323 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.822507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.822472 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1ce7a7\"" Nov 03 09:16:38.822700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.822523 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1ce7a7-dockercfg-jlpgh\"" Nov 03 09:16:38.826899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.826861 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:16:38.831950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.831915 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qhs\" (UniqueName: \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.832106 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.832048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.933433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.933403 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qhs\" (UniqueName: \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.933587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.933492 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.934127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.934108 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:38.949556 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.949533 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:16:38.953542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.953517 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:38.992595 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:38.992563 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qhs\" (UniqueName: \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\") pod \"payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:39.011528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.011459 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:16:39.034911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.034884 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.035111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.035092 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8b4\" (UniqueName: \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.099171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.087911 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:39.145329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.145291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.145544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.145527 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8b4\" (UniqueName: \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.149411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.146710 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.208166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.207711 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8b4\" (UniqueName: \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\") pod \"payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.268115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.267107 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:39.550842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.549026 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:16:39.640080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.639979 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:39.718337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:39.718311 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:16:40.257933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.257891 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" event={"ID":"e76c296c-07b4-416e-86e5-1f03e0b82fd5","Type":"ContainerStarted","Data":"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a"} Nov 03 09:16:40.276618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.276589 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" event={"ID":"e76c296c-07b4-416e-86e5-1f03e0b82fd5","Type":"ContainerStarted","Data":"07acbb52180bd7a7709eeda7ee8db731253380b81c548d026caac5fb31fa7e66"} Nov 03 09:16:40.276778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.276763 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:40.302169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.296722 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" event={"ID":"97c47e7a-2448-4556-9391-eb58d4467ff5","Type":"ContainerStarted","Data":"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7"} Nov 03 09:16:40.302432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.302406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" event={"ID":"97c47e7a-2448-4556-9391-eb58d4467ff5","Type":"ContainerStarted","Data":"dca575749ed63b461ccc6886574db068f4ce628ecb5836dbea5abef2afd5a417"} Nov 03 09:16:40.341554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.340523 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:40.728079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.728017 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" podStartSLOduration=2.72799726 podStartE2EDuration="2.72799726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:40.72057743 +0000 UTC m=+2224.388618168" watchObservedRunningTime="2026-11-03 09:16:40.72799726 +0000 UTC m=+2224.396037998" Nov 03 09:16:40.729286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:40.729240 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" podStartSLOduration=2.729226879 podStartE2EDuration="2.729226879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:40.406769945 +0000 UTC m=+2224.074810685" watchObservedRunningTime="2026-11-03 09:16:40.729226879 +0000 UTC m=+2224.397267618" Nov 03 09:16:42.948368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:42.948328 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:45.452963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:45.452931 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:46.764855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:46.764812 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:49.772172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:49.772046 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:16:49.772689 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:49.772384 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" podUID="58344624-e496-48e4-8815-c5da35945f06" containerName="maas-api" containerID="cri-o://681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c" gracePeriod=30 Nov 03 09:16:49.838505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:49.838476 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:50.078816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.078789 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:16:50.171652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.171616 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\") pod \"58344624-e496-48e4-8815-c5da35945f06\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " Nov 03 09:16:50.171842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.171735 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\" (UniqueName: \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\") pod \"58344624-e496-48e4-8815-c5da35945f06\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " Nov 03 09:16:50.171842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.171762 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\") pod \"58344624-e496-48e4-8815-c5da35945f06\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " Nov 03 09:16:50.171842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.171788 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") pod \"58344624-e496-48e4-8815-c5da35945f06\" (UID: \"58344624-e496-48e4-8815-c5da35945f06\") " Nov 03 09:16:50.172225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.172196 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca" pod "58344624-e496-48e4-8815-c5da35945f06" (UID: "58344624-e496-48e4-8815-c5da35945f06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:16:50.172503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.172464 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58344624-e496-48e4-8815-c5da35945f06-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:50.174098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.174067 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs" pod "58344624-e496-48e4-8815-c5da35945f06" (UID: "58344624-e496-48e4-8815-c5da35945f06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:50.174245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.174206 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls" pod "58344624-e496-48e4-8815-c5da35945f06" (UID: "58344624-e496-48e4-8815-c5da35945f06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:16:50.174687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.174662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt" pod "58344624-e496-48e4-8815-c5da35945f06" (UID: "58344624-e496-48e4-8815-c5da35945f06"). InnerVolumeSpecName "kube-api-access-knnjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:16:50.273905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.273867 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:50.273905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.273903 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knnjt\" (UniqueName: \"kubernetes.io/projected/58344624-e496-48e4-8815-c5da35945f06-kube-api-access-knnjt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:50.273905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.273915 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58344624-e496-48e4-8815-c5da35945f06-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:16:50.411392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.411360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:50.428613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.428538 4543 generic.go:356] "Generic (PLEG): container finished" podID="58344624-e496-48e4-8815-c5da35945f06" containerID="681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c" exitCode=0 Nov 03 09:16:50.429011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.428588 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" event={"ID":"58344624-e496-48e4-8815-c5da35945f06","Type":"ContainerDied","Data":"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c"} Nov 03 09:16:50.429011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.428830 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" event={"ID":"58344624-e496-48e4-8815-c5da35945f06","Type":"ContainerDied","Data":"fbba1c36cc374bebc15f1d1722c978d598087dd95bf94f7307461cb826679a39"} Nov 03 09:16:50.429011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.428959 4543 scope.go:122] "RemoveContainer" containerID="681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c" Nov 03 09:16:50.429660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.429426 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" Nov 03 09:16:50.447856 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.443999 4543 scope.go:122] "RemoveContainer" containerID="681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c" Nov 03 09:16:50.447856 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:16:50.444344 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c\": container with ID starting with 681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c not found: ID does not exist" containerID="681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c" Nov 03 09:16:50.447856 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.444376 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c"} err="failed to get container status \"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c\": rpc error: code = NotFound desc = could not find container \"681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c\": container with ID starting with 681bf02b93af080e7c2d9b616bf1ae5616289a6c584ee688bf6aa87c4d67e24c not found: ID does not exist" Nov 03 09:16:50.544505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.544471 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:50.599231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.599205 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:16:50.605027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:50.605000 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8"] Nov 03 09:16:51.374543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:51.374514 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:16:51.374941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:51.374573 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:16:51.861606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:51.861575 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:52.450483 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:52.450453 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58344624-e496-48e4-8815-c5da35945f06" path="/var/lib/kubelet/pods/58344624-e496-48e4-8815-c5da35945f06/volumes" Nov 03 09:16:53.010236 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.010194 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-121ee6-cbd88bccd-27lk8" podUID="58344624-e496-48e4-8815-c5da35945f06" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.86:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:16:53.897989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.897957 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:16:53.898526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.898511 4543 cpu_manager.go:414] "containerMap: removing container" podUID="58344624-e496-48e4-8815-c5da35945f06" containerName="maas-api" Nov 03 09:16:53.898526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.898527 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58344624-e496-48e4-8815-c5da35945f06" containerName="maas-api" Nov 03 09:16:53.898612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.898603 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="58344624-e496-48e4-8815-c5da35945f06" containerName="maas-api" Nov 03 09:16:53.903115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.903100 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:53.980236 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:53.980206 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:54.018869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.018820 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.018869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.018868 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.018895 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzbl\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.018955 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.019080 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.019118 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.019154 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.019192 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.019351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.019221 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.023348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.023322 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:16:54.120313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120279 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120378 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120412 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120430 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120567 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120628 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120706 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120754 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120787 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzbl\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.120976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.120913 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.121217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.121196 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.121493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.121473 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.123182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.123099 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.123398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.123242 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.183787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.183712 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.184040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.184020 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzbl\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.217945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.217923 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:54.645038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.645010 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:16:54.645730 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:16:54.645697 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b6cff4_be1b_47e3_abfb_0bb2448ad226.slice/crio-3db8f5ed318003234a1ca2b9afc42e3d54f718f8ff6fb68b3a11b50a377f72a7 WatchSource:0}: Error finding container 3db8f5ed318003234a1ca2b9afc42e3d54f718f8ff6fb68b3a11b50a377f72a7: Status 404 returned error can't find the container with id 3db8f5ed318003234a1ca2b9afc42e3d54f718f8ff6fb68b3a11b50a377f72a7 Nov 03 09:16:54.648027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.647996 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:54.648107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.648054 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:54.648107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:54.648085 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:16:55.459011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:55.458982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" event={"ID":"79b6cff4-be1b-47e3-abfb-0bb2448ad226","Type":"ContainerStarted","Data":"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34"} Nov 03 09:16:55.459406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:55.459028 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" event={"ID":"79b6cff4-be1b-47e3-abfb-0bb2448ad226","Type":"ContainerStarted","Data":"3db8f5ed318003234a1ca2b9afc42e3d54f718f8ff6fb68b3a11b50a377f72a7"} Nov 03 09:16:55.679197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:55.679133 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podStartSLOduration=2.6791195070000002 podStartE2EDuration="2.679119507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:55.628557426 +0000 UTC m=+2239.296598165" watchObservedRunningTime="2026-11-03 09:16:55.679119507 +0000 UTC m=+2239.347160245" Nov 03 09:16:56.001572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:56.000812 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:56.218178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:56.218121 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:16:56.219533 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:56.219514 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:16:56.219610 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:56.219566 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:16:57.181688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:57.181653 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:16:57.218303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:57.218274 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:16:57.218463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:57.218316 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:16:58.218615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:58.218580 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:16:58.219028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:58.218629 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:16:59.218934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:59.218902 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:16:59.219332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:16:59.218954 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:17:00.218585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:00.218551 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:17:00.218804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:00.218612 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:17:01.108375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:01.108335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:01.221663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:01.221624 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 09:17:01.221959 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:01.221927 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 09:17:01.260117 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:01.260074 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:02.223458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.223432 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:17:02.223852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.223834 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:17:02.224548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.224528 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:17:02.551795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.551691 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:02.788978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.788949 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:02.922282 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:02.922250 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:03.172050 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:03.172014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:03.355001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:03.354914 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:03.534626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:03.534591 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:04.387228 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:04.387185 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:05.560389 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:17:05.560352 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:17:07.457330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:07.457301 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:09.484191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:09.484129 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:10.237314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:10.237265 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:11.273314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:11.273283 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:15.706017 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:15.705979 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:15.836031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:15.835994 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:16.197968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:16.197937 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:16.366315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:16.366282 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:16.502223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:16.502106 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:17.602188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:17.602157 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:18.100704 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:18.100672 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:18.916098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:18.916064 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:21.836130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:21.836099 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:25.000660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:25.000624 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:25.151773 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:25.151746 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:26.066954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:26.066923 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:27.227701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:27.227670 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:31.224407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:31.224372 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:31.893129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:31.893095 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:17:31.893336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:31.893314 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" containerID="cri-o://b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b" gracePeriod=30 Nov 03 09:17:31.919088 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:31.919020 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.89:8443/health\": dial tcp 10.134.0.89:8443: connect: connection refused" Nov 03 09:17:31.981655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:31.981591 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:32.176168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.176126 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:17:32.252862 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.252835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\") pod \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " Nov 03 09:17:32.253315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.252898 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\" (UniqueName: \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\") pod \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " Nov 03 09:17:32.253315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.252976 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") pod \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " Nov 03 09:17:32.253315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.253031 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\") pod \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\" (UID: \"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa\") " Nov 03 09:17:32.253315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.253246 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca" pod "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" (UID: "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:17:32.253495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.253394 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:32.255186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.255161 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls" pod "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" (UID: "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:17:32.255186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.255171 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs" pod "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" (UID: "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:17:32.255303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.255271 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x" pod "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" (UID: "cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa"). InnerVolumeSpecName "kube-api-access-cs55x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:17:32.354741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.354713 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs55x\" (UniqueName: \"kubernetes.io/projected/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-kube-api-access-cs55x\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:32.354741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.354740 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:32.354948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.354754 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:17:32.704204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.704170 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:32.853080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.852975 4543 generic.go:356] "Generic (PLEG): container finished" podID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerID="b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b" exitCode=0 Nov 03 09:17:32.853080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.853040 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" Nov 03 09:17:32.853349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.853093 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerDied","Data":"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b"} Nov 03 09:17:32.853349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.853156 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf" event={"ID":"cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa","Type":"ContainerDied","Data":"9e90591fe376334d4256d2535077e65f2b8d2803543692bf7c212b78b6c71cd2"} Nov 03 09:17:32.853349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.853188 4543 scope.go:122] "RemoveContainer" containerID="b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b" Nov 03 09:17:32.865095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.865072 4543 scope.go:122] "RemoveContainer" containerID="e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646" Nov 03 09:17:32.874912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.874883 4543 scope.go:122] "RemoveContainer" containerID="b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b" Nov 03 09:17:32.875214 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:17:32.875193 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b\": container with ID starting with b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b not found: ID does not exist" containerID="b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b" Nov 03 09:17:32.875285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.875222 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b"} err="failed to get container status \"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b\": rpc error: code = NotFound desc = could not find container \"b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b\": container with ID starting with b51ca2977aab2024920621c2a3a618ebadfeb8b0b4ebb78f09c81f95cdad419b not found: ID does not exist" Nov 03 09:17:32.875285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.875241 4543 scope.go:122] "RemoveContainer" containerID="e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646" Nov 03 09:17:32.875502 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:17:32.875489 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646\": container with ID starting with e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646 not found: ID does not exist" containerID="e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646" Nov 03 09:17:32.875546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:32.875506 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646"} err="failed to get container status \"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646\": rpc error: code = NotFound desc = could not find container \"e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646\": container with ID starting with e4dfcb43b72988bfd159e4435564597b5479e992f07c8f40de5eb36f0691c646 not found: ID does not exist" Nov 03 09:17:33.042165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:33.033431 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:17:33.078813 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:33.078780 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0eecfd-8544f74c5c-bvdxf"] Nov 03 09:17:33.975558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:33.975527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:34.459569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:34.458938 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" path="/var/lib/kubelet/pods/cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa/volumes" Nov 03 09:17:35.748204 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:17:35.747515 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:17:36.198277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:36.198224 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:36.217167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:36.215247 4543 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:17:36.217167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:36.215312 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:36.465218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:36.464800 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:36.682159 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:36.681246 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:17:39.767742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:39.767703 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:40.033897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:40.033818 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:40.167960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:40.167928 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:41.613264 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:41.613233 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:42.104373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:42.104293 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:43.286521 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:43.286464 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:45.278470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:45.278438 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:46.605328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:46.605295 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:47.116994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:47.116956 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:47.261809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:47.261777 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:47.613786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:47.613755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:50.593217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:50.593184 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:51.712071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:51.712035 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:51.896450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:51.896406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:52.044331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:52.044245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:54.372742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.372707 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:17:54.373318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373302 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.373371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373320 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.373371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373339 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.373371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373343 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.373476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373461 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.373476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.373471 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2f6e49-50aa-48ae-b01d-e7b1c06d19fa" containerName="maas-api" Nov 03 09:17:54.378417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.378397 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454301 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454356 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454490 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454515 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454539 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454645 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454668 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454685 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxskj\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.454754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.454714 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.485129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.485095 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:17:54.555548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555516 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555557 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxskj\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555766 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555923 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.555975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.555952 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556025 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556076 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556114 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556212 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556479 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.556721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.556705 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.557985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.557967 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.558221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.558203 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.615056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.615027 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.615192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.615121 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxskj\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.691208 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.691125 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:54.985403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:54.985314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:55.122096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:55.122065 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:17:55.123320 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:17:55.123288 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cf8c22_e780_454b_8762_f58becc82ede.slice/crio-e347bbb3370e8c13db3e7cfebebbd5b307da2dc6404327eea3184e5f87f4fbae WatchSource:0}: Error finding container e347bbb3370e8c13db3e7cfebebbd5b307da2dc6404327eea3184e5f87f4fbae: Status 404 returned error can't find the container with id e347bbb3370e8c13db3e7cfebebbd5b307da2dc6404327eea3184e5f87f4fbae Nov 03 09:17:55.129637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:55.129601 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:17:55.129734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:55.129680 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:17:55.129734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:55.129711 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:17:55.169903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:55.169873 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:56.111416 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.111381 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" event={"ID":"57cf8c22-e780-454b-8762-f58becc82ede","Type":"ContainerStarted","Data":"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73"} Nov 03 09:17:56.111823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.111438 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" event={"ID":"57cf8c22-e780-454b-8762-f58becc82ede","Type":"ContainerStarted","Data":"e347bbb3370e8c13db3e7cfebebbd5b307da2dc6404327eea3184e5f87f4fbae"} Nov 03 09:17:56.166631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.166602 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:56.698183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.695423 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:17:56.702161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.699748 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:17:56.702161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:56.699808 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:17:57.695558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:57.695521 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:17:57.696176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:57.696113 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:17:58.707163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:58.706524 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:17:58.707163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:58.706591 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:17:59.705464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:59.704639 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:17:59.705464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:59.705059 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:17:59.705464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:17:59.705104 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:18:00.329166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:00.294997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:00.694168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:00.691751 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:18:00.694168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:00.691814 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:18:01.024299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:01.024209 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:01.439326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:01.439287 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:01.692039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:01.691964 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 09:18:01.692039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:01.692019 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 09:18:03.017788 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:03.017756 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:18:03.019873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:03.019850 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:18:03.033077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:03.033047 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:18:03.245131 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:03.245070 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podStartSLOduration=9.245050895 podStartE2EDuration="9.245050895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:56.232313967 +0000 UTC m=+2299.900354707" watchObservedRunningTime="2026-11-03 09:18:03.245050895 +0000 UTC m=+2306.913091635" Nov 03 09:18:04.787525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:04.787483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:04.917750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:04.917709 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:05.072083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:05.072002 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:05.561241 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:18:05.561207 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:18:05.564809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:05.564785 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:06.085307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.085272 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:18:06.089243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.089227 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.153217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.151535 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-465707-openshift-default-dockercfg-f4fpx\"" Nov 03 09:18:06.196328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196152 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196561 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196622 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196774 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196803 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196827 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.196874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196857 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgc88\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.197091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196908 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.197091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.196975 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.202791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.202716 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:06.208876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.207015 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:18:06.298792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.298758 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300489 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300520 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgc88\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300572 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300661 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300743 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300812 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.300868 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.301871 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.302350 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.302614 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.303571 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.304762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.304677 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.306889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.306829 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.310085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.310064 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.361869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.361836 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgc88\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.362467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.362444 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\") pod \"e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.402614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.402586 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:06.504748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.504711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:06.632092 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.632010 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:06.839022 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.838999 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:18:06.839434 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:18:06.839411 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efb044f_7641_439b_97d4_ad5bd9928238.slice/crio-c5e672786411ca1dd6f5e36c2a1a0ff0c9071fd1d6040637388ca5e693ce9031 WatchSource:0}: Error finding container c5e672786411ca1dd6f5e36c2a1a0ff0c9071fd1d6040637388ca5e693ce9031: Status 404 returned error can't find the container with id c5e672786411ca1dd6f5e36c2a1a0ff0c9071fd1d6040637388ca5e693ce9031 Nov 03 09:18:06.841872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.841844 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:06.841939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.841907 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:06.841974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:06.841938 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:07.221966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.221870 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerStarted","Data":"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66"} Nov 03 09:18:07.221966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.221927 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerStarted","Data":"c5e672786411ca1dd6f5e36c2a1a0ff0c9071fd1d6040637388ca5e693ce9031"} Nov 03 09:18:07.344188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.344124 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podStartSLOduration=1.344111158 podStartE2EDuration="1.344111158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:07.340776234 +0000 UTC m=+2311.008816980" watchObservedRunningTime="2026-11-03 09:18:07.344111158 +0000 UTC m=+2311.012151948" Nov 03 09:18:07.393592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.393562 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:07.403503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.403483 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:07.409259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.409239 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:07.576366 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:07.576291 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.099750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.099712 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:18:08.103832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.103817 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.122730 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.122692 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.122870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.122852 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.122981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.122959 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123007 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123083 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123160 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123120 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123209 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hkg\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123265 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123250 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.123360 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.123343 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.138534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.138505 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.167501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.165760 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-6f8ed0-openshift-default-dockercfg-d7dwc\"" Nov 03 09:18:08.171804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.171776 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:18:08.225096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225066 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225206 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225348 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225416 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225445 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225471 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225500 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225550 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hkg\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225579 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.225995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.225973 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.226188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.226085 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.226373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.226351 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.226630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.226611 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.226849 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.226825 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.228431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.228406 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.228721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.228703 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.229344 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.229327 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:08.230395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.230379 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:18:08.284270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.284228 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.284913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.284891 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.285128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.285112 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hkg\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\") pod \"e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.418242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.418212 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:08.424327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.424290 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.744745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.744646 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.885090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.885054 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:08.898986 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.898961 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:18:08.934975 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:18:08.934946 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0def2f1_9f7a_426a_b105_03e220428240.slice/crio-9d3b367109f7ed1e67e0a1cff18e91c2e465d4b800cd9848e8da68dc0851b0fd WatchSource:0}: Error finding container 9d3b367109f7ed1e67e0a1cff18e91c2e465d4b800cd9848e8da68dc0851b0fd: Status 404 returned error can't find the container with id 9d3b367109f7ed1e67e0a1cff18e91c2e465d4b800cd9848e8da68dc0851b0fd Nov 03 09:18:08.937437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.937405 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:08.937512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.937477 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:08.937512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:08.937506 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:09.237606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:09.237563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerStarted","Data":"dbf9e88bb878ec15a0a4416657d1908c3907cfe9f71b469eecc91262589f6206"} Nov 03 09:18:09.238021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:09.237649 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerStarted","Data":"9d3b367109f7ed1e67e0a1cff18e91c2e465d4b800cd9848e8da68dc0851b0fd"} Nov 03 09:18:09.359283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:09.359185 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podStartSLOduration=1.35916837 podStartE2EDuration="1.35916837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:09.356828766 +0000 UTC m=+2313.024869511" watchObservedRunningTime="2026-11-03 09:18:09.35916837 +0000 UTC m=+2313.027209100" Nov 03 09:18:09.418389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:09.418363 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:10.089442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:10.089406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:10.270391 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:10.270344 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:10.438159 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:10.425566 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:10.461600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:10.461423 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:11.250003 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:11.249965 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:11.250784 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:11.250764 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:18:12.859392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:12.859327 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:13.010583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:13.010499 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:13.150255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:13.150203 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:15.552953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:15.552888 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:15.755313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:15.755280 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:18.127350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:18.127316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:18.314379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:18.314335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:18.704670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:18.704625 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:23.549406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.549374 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:18:23.553389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.553375 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.563822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.563790 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.563928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.563828 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248lc\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.564023 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.563999 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.564058 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.564039 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.564099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.564065 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.564239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.564222 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.564314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.564302 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.607865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.607831 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 09:18:23.607865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.607851 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 09:18:23.608074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.607959 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 09:18:23.657696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.657673 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9gqlj\"" Nov 03 09:18:23.661517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.661492 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:18:23.665319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665297 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665328 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665352 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665406 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665525 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.665607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.665553 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248lc\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.666132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.666102 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.668433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.668394 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.668595 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.668576 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.668777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.668758 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.669472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.669453 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.724917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.724894 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.725083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.725065 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248lc\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-blw95\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:23.865458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:23.865422 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:24.235200 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:24.235090 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:24.304346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:24.304310 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:18:24.305799 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:18:24.305763 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda189e9e5_976c_46ee_b2ed_19adc708f803.slice/crio-dbb4f1ebbb1407454e3bae6dc854c2d0d3d4105849fb94e6110ae3bffdcc2ff0 WatchSource:0}: Error finding container dbb4f1ebbb1407454e3bae6dc854c2d0d3d4105849fb94e6110ae3bffdcc2ff0: Status 404 returned error can't find the container with id dbb4f1ebbb1407454e3bae6dc854c2d0d3d4105849fb94e6110ae3bffdcc2ff0 Nov 03 09:18:24.308414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:24.308379 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:24.308517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:24.308455 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:18:24.331589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:24.331562 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" event={"ID":"a189e9e5-976c-46ee-b2ed-19adc708f803","Type":"ContainerStarted","Data":"dbb4f1ebbb1407454e3bae6dc854c2d0d3d4105849fb94e6110ae3bffdcc2ff0"} Nov 03 09:18:25.339202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:25.339165 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" event={"ID":"a189e9e5-976c-46ee-b2ed-19adc708f803","Type":"ContainerStarted","Data":"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62"} Nov 03 09:18:25.339683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:25.339304 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:25.341464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:25.341436 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-blw95 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:18:25.341614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:25.341493 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:18:25.467181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:25.467100 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" podStartSLOduration=2.46708177 podStartE2EDuration="2.46708177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:25.463204643 +0000 UTC m=+2329.131245383" watchObservedRunningTime="2026-11-03 09:18:25.46708177 +0000 UTC m=+2329.135122509" Nov 03 09:18:26.345909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:26.345882 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:18:29.826331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:29.826299 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:31.222456 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:31.222425 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:35.265941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:35.265906 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:35.562298 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:18:35.562199 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:18:40.252132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:40.252095 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:43.636164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:43.635648 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:44.759954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:44.759919 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:45.453697 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:45.451780 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:45.594554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:45.594487 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:45.986314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:45.986283 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:47.531622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:47.531594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:49.961417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:49.961381 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:50.146336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:50.146278 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:50.281833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:50.281755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:50.730179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:50.730147 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:50.885111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:50.885079 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:51.152883 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:51.152855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:54.015701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:54.015661 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:54.200877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:54.200837 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:18:56.552067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:18:56.552033 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:01.811367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:01.811334 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:02.084056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:02.083980 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:02.222955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:02.222925 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:04.479444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:04.479413 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:04.606814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:04.606773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:05.219337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:05.219300 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:05.560274 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:19:05.560193 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:19:07.325802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.325772 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:19:07.329656 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.329640 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.490719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.490686 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-6f8ed0-serving-cert\"" Nov 03 09:19:07.500832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.500794 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.501011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.500844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.501011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.500918 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bhp\" (UniqueName: \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.501123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.501034 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.501278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.501225 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:19:07.532398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.532367 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:07.602997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.602910 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.603543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.603517 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.603776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.603762 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bhp\" (UniqueName: \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.604036 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.604011 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.604929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.604423 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.604929 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:19:07.604826 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-serving-cert: secret "maas-api-e2e-praxis-b-6f8ed0-serving-cert" not found Nov 03 09:19:07.605420 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:19:07.605194 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls podName:977a22bf-981e-40ae-ad5b-51b82a2443e9 nodeName:}" failed. No retries permitted until 2026-11-03 09:19:08.105165757 +0000 UTC m=+2371.773206488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls") pod "maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" (UID: "977a22bf-981e-40ae-ad5b-51b82a2443e9") : secret "maas-api-e2e-praxis-b-6f8ed0-serving-cert" not found Nov 03 09:19:07.609896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.609871 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.668099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.668062 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bhp\" (UniqueName: \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:07.747375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.747348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:07.880900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:07.880868 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:08.114376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:08.114345 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:08.116842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:08.116821 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:08.245888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:08.245811 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:08.622294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:08.622263 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:19:08.622708 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:19:08.622434 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977a22bf_981e_40ae_ad5b_51b82a2443e9.slice/crio-8cac946674c69d0b531a843ce0f40b27c91668afd5a5eb1fd5638bc0db02078a WatchSource:0}: Error finding container 8cac946674c69d0b531a843ce0f40b27c91668afd5a5eb1fd5638bc0db02078a: Status 404 returned error can't find the container with id 8cac946674c69d0b531a843ce0f40b27c91668afd5a5eb1fd5638bc0db02078a Nov 03 09:19:08.753725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:08.753692 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" event={"ID":"977a22bf-981e-40ae-ad5b-51b82a2443e9","Type":"ContainerStarted","Data":"8cac946674c69d0b531a843ce0f40b27c91668afd5a5eb1fd5638bc0db02078a"} Nov 03 09:19:09.762466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:09.762371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" event={"ID":"977a22bf-981e-40ae-ad5b-51b82a2443e9","Type":"ContainerStarted","Data":"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc"} Nov 03 09:19:09.762854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:09.762503 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:09.881577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:09.881526 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" podStartSLOduration=2.048377628 podStartE2EDuration="2.881512737s" totalImagesPullingTime="833.135109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:09.879310999 +0000 UTC m=+2373.547351738" watchObservedRunningTime="2026-11-03 09:19:09.881512737 +0000 UTC m=+2373.549553475" Nov 03 09:19:13.108361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:13.108325 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:15.796059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:15.796033 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:19:18.272489 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:18.272406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:23.239558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:23.239515 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:28.346835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:28.346798 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:33.443975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:33.443935 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:34.993900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.993867 4543 generic.go:356] "Generic (PLEG): container finished" podID="b0def2f1-9f7a-426a-b105-03e220428240" containerID="dbf9e88bb878ec15a0a4416657d1908c3907cfe9f71b469eecc91262589f6206" exitCode=0 Nov 03 09:19:34.994351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.993907 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerDied","Data":"dbf9e88bb878ec15a0a4416657d1908c3907cfe9f71b469eecc91262589f6206"} Nov 03 09:19:34.994351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.994314 4543 scope.go:122] "RemoveContainer" containerID="dbf9e88bb878ec15a0a4416657d1908c3907cfe9f71b469eecc91262589f6206" Nov 03 09:19:34.995723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.995691 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:34.995845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.995768 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:34.995845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:34.995815 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:35.560757 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:19:35.560718 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:19:36.012388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.012360 4543 generic.go:356] "Generic (PLEG): container finished" podID="1efb044f-7641-439b-97d4-ad5bd9928238" containerID="f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66" exitCode=0 Nov 03 09:19:36.012883 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.012430 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerDied","Data":"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66"} Nov 03 09:19:36.012947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.012889 4543 scope.go:122] "RemoveContainer" containerID="f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66" Nov 03 09:19:36.014610 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.014579 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:36.014610 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.014598 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerStarted","Data":"57d14d0ec156ab788499ee17d2c3317ea6904fa411de99ee84f6702aa3d9cd72"} Nov 03 09:19:36.014782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.014654 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:36.014782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.014689 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:19:36.418977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.418942 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:19:36.419191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.419132 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 09:19:36.419281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.419199 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 09:19:36.771126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.771041 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:19:36.775507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:36.775485 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:19:37.043277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.043185 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerStarted","Data":"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81"} Nov 03 09:19:37.409164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.408552 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:19:37.409164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.408733 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 09:19:37.409164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.408785 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 09:19:37.431214 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.429548 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 09:19:37.431214 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:37.429608 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 09:19:38.230244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.229230 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:19:38.413387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.413344 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 09:19:38.413599 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.413410 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 09:19:38.421172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.421118 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 09:19:38.421347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.421187 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 09:19:38.680156 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.680096 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:38.820736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:38.820681 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:39.120224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.104544 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:39.419218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.418250 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 09:19:39.419218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.418309 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 09:19:39.423163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.420512 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 09:19:39.423163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.420557 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 09:19:39.826845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.826761 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:39.978803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:39.978758 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:40.422162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:40.421266 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 09:19:40.422162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:40.421345 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 09:19:40.427161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:40.425819 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 09:19:40.427161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:40.425868 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 09:19:41.250539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:41.250496 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:19:41.407161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:41.406499 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 09:19:41.407161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:41.406558 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 09:19:41.423262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:41.423234 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:19:42.089737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:42.089710 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:19:42.407489 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:42.407460 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:19:42.408483 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:42.408463 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:19:43.985775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:43.985741 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:44.124385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:44.124352 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:44.263466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:44.263377 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:44.432550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:44.432521 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:47.550623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:47.550588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:47.740249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:47.740200 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:47.888000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:47.887898 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:49.193099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:49.193064 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:49.351711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:49.351680 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:49.485791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:49.485693 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:54.432689 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:54.432644 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:19:59.554056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:19:59.554012 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:01.913572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:01.913540 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:02.385193 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:02.385164 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:02.692466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:02.692381 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:03.502331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.502292 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:20:03.505107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.505092 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.560199 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.560167 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b196c8-openshift-default-dockercfg-hrsmg\"" Nov 03 09:20:03.592947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.592911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:03.613591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.613560 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660417 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660604 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660647 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660727 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660840 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660874 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660924 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4bh\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.671444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.660951 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.762415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.762326 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.762748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.762717 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.762853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.762832 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.762903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.762884 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.762946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.762934 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763024 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763008 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763058 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763043 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763190 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763237 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763291 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4bh\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763335 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.763867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.763844 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.764091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.764072 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.764427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.764400 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.765858 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.765838 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.766645 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.766618 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.776900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.774676 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:03.824171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.824088 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:03.824387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:03.824337 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4bh\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:04.116914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:04.116883 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:04.554712 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:04.554688 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:20:04.555420 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:20:04.555398 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe29adb_8c7b_4eee_8a75_5502439d5978.slice/crio-391a5421b794cf239070b063202b10447dad6e5017528d4fdbf59ec9ad872fcc WatchSource:0}: Error finding container 391a5421b794cf239070b063202b10447dad6e5017528d4fdbf59ec9ad872fcc: Status 404 returned error can't find the container with id 391a5421b794cf239070b063202b10447dad6e5017528d4fdbf59ec9ad872fcc Nov 03 09:20:04.557997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:04.557968 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:20:04.558100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:04.558050 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:20:04.558100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:04.558082 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:20:05.076350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:05.076318 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:05.304980 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:05.304943 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerStarted","Data":"d68a4ff8f719418e8f2ac5f84538480e6889a7578089c10dec2d99e31f77ad70"} Nov 03 09:20:05.305253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:05.304993 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerStarted","Data":"391a5421b794cf239070b063202b10447dad6e5017528d4fdbf59ec9ad872fcc"} Nov 03 09:20:05.491820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:05.491760 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podStartSLOduration=2.491745529 podStartE2EDuration="2.491745529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:05.474781169 +0000 UTC m=+2429.142821910" watchObservedRunningTime="2026-11-03 09:20:05.491745529 +0000 UTC m=+2429.159786263" Nov 03 09:20:05.560630 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:20:05.560587 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:20:05.575373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:05.575323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:06.117971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:06.117938 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:06.123966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:06.123941 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:06.311164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:06.311108 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:06.312040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:06.312020 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:20:08.239872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:08.239840 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:08.396350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:08.396311 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:10.633661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:10.633625 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:12.189686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:12.189647 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:12.639555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:12.639524 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:13.295161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:13.276615 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:13.618880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:13.618840 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:13.765667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:13.765621 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:13.907512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:13.907435 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:16.285130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:16.285093 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:18.575172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:18.575122 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:18.761781 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:18.761749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:18.910725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:18.910691 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:19.278576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:19.278492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:20.821454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:20.821417 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:21.912342 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:21.912313 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:23.072555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:23.072509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:26.004032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:26.003992 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:26.135281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:26.135246 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:27.052732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:27.052701 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:20:27.169759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:27.169715 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:20:27.248201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:27.248171 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:27.419420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:27.419379 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:27.552448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:27.552414 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:28.339277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:28.339245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:32.053516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.053454 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" podUID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" containerName="payload-processing" containerID="cri-o://44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a" gracePeriod=25 Nov 03 09:20:32.170967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.170934 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" podUID="97c47e7a-2448-4556-9391-eb58d4467ff5" containerName="payload-pre-processing" containerID="cri-o://eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7" gracePeriod=25 Nov 03 09:20:32.322526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.322450 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:20:32.399760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.399728 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\") pod \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " Nov 03 09:20:32.400026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.399781 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\" (UniqueName: \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\") pod \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\" (UID: \"e76c296c-07b4-416e-86e5-1f03e0b82fd5\") " Nov 03 09:20:32.400478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.400452 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume" pod "e76c296c-07b4-416e-86e5-1f03e0b82fd5" (UID: "e76c296c-07b4-416e-86e5-1f03e0b82fd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:20:32.402729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.402704 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs" pod "e76c296c-07b4-416e-86e5-1f03e0b82fd5" (UID: "e76c296c-07b4-416e-86e5-1f03e0b82fd5"). InnerVolumeSpecName "kube-api-access-65qhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:20:32.432074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.432055 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:20:32.497792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.497758 4543 generic.go:356] "Generic (PLEG): container finished" podID="97c47e7a-2448-4556-9391-eb58d4467ff5" containerID="eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7" exitCode=0 Nov 03 09:20:32.497945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.497805 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" Nov 03 09:20:32.497945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.497839 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" event={"ID":"97c47e7a-2448-4556-9391-eb58d4467ff5","Type":"ContainerDied","Data":"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7"} Nov 03 09:20:32.497945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.497894 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2" event={"ID":"97c47e7a-2448-4556-9391-eb58d4467ff5","Type":"ContainerDied","Data":"dca575749ed63b461ccc6886574db068f4ce628ecb5836dbea5abef2afd5a417"} Nov 03 09:20:32.497945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.497917 4543 scope.go:122] "RemoveContainer" containerID="eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7" Nov 03 09:20:32.499402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.499377 4543 generic.go:356] "Generic (PLEG): container finished" podID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" containerID="44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a" exitCode=0 Nov 03 09:20:32.499529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.499417 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" Nov 03 09:20:32.499529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.499462 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" event={"ID":"e76c296c-07b4-416e-86e5-1f03e0b82fd5","Type":"ContainerDied","Data":"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a"} Nov 03 09:20:32.499529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.499516 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw" event={"ID":"e76c296c-07b4-416e-86e5-1f03e0b82fd5","Type":"ContainerDied","Data":"07acbb52180bd7a7709eeda7ee8db731253380b81c548d026caac5fb31fa7e66"} Nov 03 09:20:32.501539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.501517 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\") pod \"97c47e7a-2448-4556-9391-eb58d4467ff5\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " Nov 03 09:20:32.501664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.501575 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\" (UniqueName: \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\") pod \"97c47e7a-2448-4556-9391-eb58d4467ff5\" (UID: \"97c47e7a-2448-4556-9391-eb58d4467ff5\") " Nov 03 09:20:32.501980 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.501957 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume" pod "97c47e7a-2448-4556-9391-eb58d4467ff5" (UID: "97c47e7a-2448-4556-9391-eb58d4467ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:20:32.502063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.502013 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e76c296c-07b4-416e-86e5-1f03e0b82fd5-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:32.502063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.502029 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qhs\" (UniqueName: \"kubernetes.io/projected/e76c296c-07b4-416e-86e5-1f03e0b82fd5-kube-api-access-65qhs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:32.503954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.503934 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4" pod "97c47e7a-2448-4556-9391-eb58d4467ff5" (UID: "97c47e7a-2448-4556-9391-eb58d4467ff5"). InnerVolumeSpecName "kube-api-access-7w8b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:20:32.508291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.508276 4543 scope.go:122] "RemoveContainer" containerID="eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7" Nov 03 09:20:32.508576 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:20:32.508556 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7\": container with ID starting with eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7 not found: ID does not exist" containerID="eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7" Nov 03 09:20:32.508679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.508580 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7"} err="failed to get container status \"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7\": rpc error: code = NotFound desc = could not find container \"eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7\": container with ID starting with eb7f270141c8903ca8cf416c3b66ae8483869930b7908779c312296a1d4355e7 not found: ID does not exist" Nov 03 09:20:32.508679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.508596 4543 scope.go:122] "RemoveContainer" containerID="44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a" Nov 03 09:20:32.517213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.517191 4543 scope.go:122] "RemoveContainer" containerID="44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a" Nov 03 09:20:32.517445 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:20:32.517425 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a\": container with ID starting with 44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a not found: ID does not exist" containerID="44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a" Nov 03 09:20:32.517499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.517452 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a"} err="failed to get container status \"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a\": rpc error: code = NotFound desc = could not find container \"44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a\": container with ID starting with 44f0373f67cf346e85340d8256811670f415bd94036c36af16a4968102b89d5a not found: ID does not exist" Nov 03 09:20:32.590939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.590863 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:32.603253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.603229 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c47e7a-2448-4556-9391-eb58d4467ff5-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:32.603253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.603249 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w8b4\" (UniqueName: \"kubernetes.io/projected/97c47e7a-2448-4556-9391-eb58d4467ff5-kube-api-access-7w8b4\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:20:32.676409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.676370 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:20:32.682525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.682487 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1ce7a7-6b49975d4c-tn4xw"] Nov 03 09:20:32.931822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.931784 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:20:32.933549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:32.933523 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1ce7a7-85f5f75d8c-k4mg2"] Nov 03 09:20:34.451909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:34.451881 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c47e7a-2448-4556-9391-eb58d4467ff5" path="/var/lib/kubelet/pods/97c47e7a-2448-4556-9391-eb58d4467ff5/volumes" Nov 03 09:20:34.452338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:34.452301 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" path="/var/lib/kubelet/pods/e76c296c-07b4-416e-86e5-1f03e0b82fd5/volumes" Nov 03 09:20:34.714041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:34.713969 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:35.561898 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:20:35.561851 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:20:37.582438 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:37.582411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:41.978249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:41.978213 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:43.487716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.487686 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:20:43.488251 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488236 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" containerName="payload-processing" Nov 03 09:20:43.488303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488251 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" containerName="payload-processing" Nov 03 09:20:43.488303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488290 4543 cpu_manager.go:414] "containerMap: removing container" podUID="97c47e7a-2448-4556-9391-eb58d4467ff5" containerName="payload-pre-processing" Nov 03 09:20:43.488303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488295 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c47e7a-2448-4556-9391-eb58d4467ff5" containerName="payload-pre-processing" Nov 03 09:20:43.488398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488371 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76c296c-07b4-416e-86e5-1f03e0b82fd5" containerName="payload-processing" Nov 03 09:20:43.488398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.488386 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c47e7a-2448-4556-9391-eb58d4467ff5" containerName="payload-pre-processing" Nov 03 09:20:43.493323 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.493307 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.597453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.597413 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-506e6d-serving-cert\"" Nov 03 09:20:43.600623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.600597 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:20:43.639476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.639445 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.639648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.639499 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.639708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.639654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfxm\" (UniqueName: \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.639868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.639847 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.673501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.673467 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:43.741414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.741346 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.741546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.741426 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.741546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.741457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.741546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.741506 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfxm\" (UniqueName: \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.741936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.741916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.743899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.743878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.743997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.743928 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.803065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.803031 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfxm\" (UniqueName: \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\") pod \"maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.806865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.806837 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:43.911181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:43.907783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:44.040397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:44.040314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:44.184686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:44.184663 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:20:44.184898 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:20:44.184876 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415508db_2301_4f55_b34e_43734a64d94d.slice/crio-7aaf31326b5f935ab5227cc7e493d086310220e739e2473ec6d041bd5e5cba28 WatchSource:0}: Error finding container 7aaf31326b5f935ab5227cc7e493d086310220e739e2473ec6d041bd5e5cba28: Status 404 returned error can't find the container with id 7aaf31326b5f935ab5227cc7e493d086310220e739e2473ec6d041bd5e5cba28 Nov 03 09:20:44.328759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:44.328685 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:44.575700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:44.575664 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" event={"ID":"415508db-2301-4f55-b34e-43734a64d94d","Type":"ContainerStarted","Data":"7aaf31326b5f935ab5227cc7e493d086310220e739e2473ec6d041bd5e5cba28"} Nov 03 09:20:45.583090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:45.583057 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" event={"ID":"415508db-2301-4f55-b34e-43734a64d94d","Type":"ContainerStarted","Data":"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0"} Nov 03 09:20:45.583510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:45.583118 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:45.704006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:45.703956 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" podStartSLOduration=1.870635705 podStartE2EDuration="2.703944135s" totalImagesPullingTime="833.30843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:45.701645431 +0000 UTC m=+2469.369686170" watchObservedRunningTime="2026-11-03 09:20:45.703944135 +0000 UTC m=+2469.371984873" Nov 03 09:20:48.252999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:48.252915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:48.900264 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:48.900221 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:50.895804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:50.895770 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:51.038132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:51.038104 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:51.595324 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:51.595295 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:20:52.531646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:52.531616 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:53.070811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:53.070779 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:53.819317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:53.819283 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:54.209303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:54.209270 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:20:59.652214 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:20:59.652183 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:00.416807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:00.416774 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:01.874718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:01.874683 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:02.013776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:02.013746 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:02.188694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:02.188582 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:02.622246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:02.622213 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:03.724578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:03.724535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:03.879251 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:03.879216 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:05.394702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:05.394668 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:05.561577 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:21:05.561534 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:21:05.908651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:05.908608 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:06.897913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:06.897883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:07.787901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:07.787869 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:07.967853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:07.967824 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:08.112823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:08.112789 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:10.828891 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:10.828857 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:11.054721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:11.054687 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:11.244950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:11.240828 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:15.786821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:15.786788 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:15.939166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:15.939105 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:16.071330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:16.071243 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:20.324040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.324009 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:21:20.324455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.324240 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" podUID="977a22bf-981e-40ae-ad5b-51b82a2443e9" containerName="maas-api" containerID="cri-o://13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc" gracePeriod=30 Nov 03 09:21:20.420314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.420278 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:20.635796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.635774 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:21:20.683505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.683476 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\") pod \"977a22bf-981e-40ae-ad5b-51b82a2443e9\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " Nov 03 09:21:20.683657 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.683531 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") pod \"977a22bf-981e-40ae-ad5b-51b82a2443e9\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " Nov 03 09:21:20.683657 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.683577 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\" (UniqueName: \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\") pod \"977a22bf-981e-40ae-ad5b-51b82a2443e9\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " Nov 03 09:21:20.683657 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.683621 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\") pod \"977a22bf-981e-40ae-ad5b-51b82a2443e9\" (UID: \"977a22bf-981e-40ae-ad5b-51b82a2443e9\") " Nov 03 09:21:20.683848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.683820 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca" pod "977a22bf-981e-40ae-ad5b-51b82a2443e9" (UID: "977a22bf-981e-40ae-ad5b-51b82a2443e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:21:20.684089 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.684065 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/977a22bf-981e-40ae-ad5b-51b82a2443e9-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:20.685765 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.685745 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls" pod "977a22bf-981e-40ae-ad5b-51b82a2443e9" (UID: "977a22bf-981e-40ae-ad5b-51b82a2443e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:21:20.685765 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.685752 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp" pod "977a22bf-981e-40ae-ad5b-51b82a2443e9" (UID: "977a22bf-981e-40ae-ad5b-51b82a2443e9"). InnerVolumeSpecName "kube-api-access-x9bhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:21:20.685890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.685780 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs" pod "977a22bf-981e-40ae-ad5b-51b82a2443e9" (UID: "977a22bf-981e-40ae-ad5b-51b82a2443e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:21:20.786905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.785418 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bhp\" (UniqueName: \"kubernetes.io/projected/977a22bf-981e-40ae-ad5b-51b82a2443e9-kube-api-access-x9bhp\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:20.786905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.785446 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:20.786905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.785455 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/977a22bf-981e-40ae-ad5b-51b82a2443e9-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:20.807314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.807272 4543 generic.go:356] "Generic (PLEG): container finished" podID="977a22bf-981e-40ae-ad5b-51b82a2443e9" containerID="13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc" exitCode=0 Nov 03 09:21:20.807498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.807471 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" event={"ID":"977a22bf-981e-40ae-ad5b-51b82a2443e9","Type":"ContainerDied","Data":"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc"} Nov 03 09:21:20.807617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.807534 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" event={"ID":"977a22bf-981e-40ae-ad5b-51b82a2443e9","Type":"ContainerDied","Data":"8cac946674c69d0b531a843ce0f40b27c91668afd5a5eb1fd5638bc0db02078a"} Nov 03 09:21:20.807617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.807566 4543 scope.go:122] "RemoveContainer" containerID="13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc" Nov 03 09:21:20.807747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.807733 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv" Nov 03 09:21:20.820812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.820792 4543 scope.go:122] "RemoveContainer" containerID="13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc" Nov 03 09:21:20.821095 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:21:20.821076 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc\": container with ID starting with 13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc not found: ID does not exist" containerID="13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc" Nov 03 09:21:20.821180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.821105 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc"} err="failed to get container status \"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc\": rpc error: code = NotFound desc = could not find container \"13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc\": container with ID starting with 13cea62e86df6f81d9fdb4a6a49147eafb2f8ee073b54dea2b17c59bb0b22fbc not found: ID does not exist" Nov 03 09:21:20.980071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.980030 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:21:20.985056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:20.985030 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6f8ed0-64475fdcdc-drrfv"] Nov 03 09:21:21.977423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:21.977394 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:21:21.978013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:21.977998 4543 cpu_manager.go:414] "containerMap: removing container" podUID="977a22bf-981e-40ae-ad5b-51b82a2443e9" containerName="maas-api" Nov 03 09:21:21.978055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:21.978014 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977a22bf-981e-40ae-ad5b-51b82a2443e9" containerName="maas-api" Nov 03 09:21:21.978176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:21.978152 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="977a22bf-981e-40ae-ad5b-51b82a2443e9" containerName="maas-api" Nov 03 09:21:21.982881 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:21.982866 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.013302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.013279 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.013409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.013348 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9jt\" (UniqueName: \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.013454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.013437 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.013505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.013491 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.046123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.046091 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-3df1cc-serving-cert\"" Nov 03 09:21:22.048628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.048587 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:21:22.115280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.115253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.115456 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.115338 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.115456 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:21:22.115399 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-serving-cert: secret "maas-api-e2e-shared-b-w1-3df1cc-serving-cert" not found Nov 03 09:21:22.115541 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:21:22.115474 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls podName:63c4dae5-7266-4d36-a027-d47fb5dcc939 nodeName:}" failed. No retries permitted until 2026-11-03 09:21:22.615452196 +0000 UTC m=+2506.283492926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls") pod "maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" (UID: "63c4dae5-7266-4d36-a027-d47fb5dcc939") : secret "maas-api-e2e-shared-b-w1-3df1cc-serving-cert" not found Nov 03 09:21:22.115541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.115518 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.115637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.115619 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9jt\" (UniqueName: \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.116019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.116002 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.118225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.118202 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.175111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.175081 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9jt\" (UniqueName: \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.188457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.188430 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:22.356072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.355980 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:22.452691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.452656 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977a22bf-981e-40ae-ad5b-51b82a2443e9" path="/var/lib/kubelet/pods/977a22bf-981e-40ae-ad5b-51b82a2443e9/volumes" Nov 03 09:21:22.496382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.496344 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:22.624091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.624060 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.626530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.626496 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:22.896826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:22.896740 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:23.274604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:23.274570 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:21:23.275019 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:21:23.274935 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c4dae5_7266_4d36_a027_d47fb5dcc939.slice/crio-48376281800afdcb1d3b50cf077dd53d411a7c5bb806d3cf6ea826dec7e35101 WatchSource:0}: Error finding container 48376281800afdcb1d3b50cf077dd53d411a7c5bb806d3cf6ea826dec7e35101: Status 404 returned error can't find the container with id 48376281800afdcb1d3b50cf077dd53d411a7c5bb806d3cf6ea826dec7e35101 Nov 03 09:21:23.279313 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:23.276964 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:21:23.512311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:23.512279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:23.646941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:23.646904 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:23.829080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:23.829043 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" event={"ID":"63c4dae5-7266-4d36-a027-d47fb5dcc939","Type":"ContainerStarted","Data":"48376281800afdcb1d3b50cf077dd53d411a7c5bb806d3cf6ea826dec7e35101"} Nov 03 09:21:24.836886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:24.836850 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" event={"ID":"63c4dae5-7266-4d36-a027-d47fb5dcc939","Type":"ContainerStarted","Data":"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c"} Nov 03 09:21:24.837398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:24.836936 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:24.958186 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:24.958099 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" podStartSLOduration=3.16502524 podStartE2EDuration="3.958084934s" totalImagesPullingTime="793.059694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:24.954497944 +0000 UTC m=+2508.622538683" watchObservedRunningTime="2026-11-03 09:21:24.958084934 +0000 UTC m=+2508.626125671" Nov 03 09:21:25.175352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:25.175317 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:25.765263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:25.765232 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:21:25.765530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:25.765504 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" containerID="cri-o://57d14d0ec156ab788499ee17d2c3317ea6904fa411de99ee84f6702aa3d9cd72" gracePeriod=30 Nov 03 09:21:25.772555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:25.772527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:27.090748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:27.090718 4543 patch_prober.go:29] interesting pod/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:21:27.091128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:27.090765 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:21:27.889025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:27.888995 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:30.905189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:30.905128 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:21:30.937294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:30.937257 4543 generic.go:356] "Generic (PLEG): container finished" podID="b0def2f1-9f7a-426a-b105-03e220428240" containerID="57d14d0ec156ab788499ee17d2c3317ea6904fa411de99ee84f6702aa3d9cd72" exitCode=0 Nov 03 09:21:30.937294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:30.937299 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerDied","Data":"57d14d0ec156ab788499ee17d2c3317ea6904fa411de99ee84f6702aa3d9cd72"} Nov 03 09:21:30.937540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:30.937367 4543 scope.go:122] "RemoveContainer" containerID="dbf9e88bb878ec15a0a4416657d1908c3907cfe9f71b469eecc91262589f6206" Nov 03 09:21:31.059699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.059668 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:21:31.134613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134583 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.134801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134641 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.134801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134764 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.134907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.134907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134868 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.135002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134932 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.135002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.134967 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.135089 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135041 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.135231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135196 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\") pod \"b0def2f1-9f7a-426a-b105-03e220428240\" (UID: \"b0def2f1-9f7a-426a-b105-03e220428240\") " Nov 03 09:21:31.135460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135341 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:21:31.135659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:21:31.135883 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135864 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:21:31.136000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135886 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:21:31.136100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.136040 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:21:31.136203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.135994 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0def2f1-9f7a-426a-b105-03e220428240-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.136306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.136293 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.137272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.137247 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "kube-api-access-s9hkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:21:31.137509 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.137481 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:21:31.137941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.137914 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:21:31.138084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.138069 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo" pod "b0def2f1-9f7a-426a-b105-03e220428240" (UID: "b0def2f1-9f7a-426a-b105-03e220428240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:21:31.237160 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237102 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237160 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237131 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237173 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237182 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9hkg\" (UniqueName: \"kubernetes.io/projected/b0def2f1-9f7a-426a-b105-03e220428240-kube-api-access-s9hkg\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237191 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237201 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0def2f1-9f7a-426a-b105-03e220428240-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.237374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.237209 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0def2f1-9f7a-426a-b105-03e220428240-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:21:31.945470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.945434 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" event={"ID":"b0def2f1-9f7a-426a-b105-03e220428240","Type":"ContainerDied","Data":"9d3b367109f7ed1e67e0a1cff18e91c2e465d4b800cd9848e8da68dc0851b0fd"} Nov 03 09:21:31.945936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.945489 4543 scope.go:122] "RemoveContainer" containerID="57d14d0ec156ab788499ee17d2c3317ea6904fa411de99ee84f6702aa3d9cd72" Nov 03 09:21:31.945936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:31.945452 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7" Nov 03 09:21:32.169148 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:32.169108 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:21:32.220683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:32.220600 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f8ed0-openshift-default-b56b68bd9-jw7g7"] Nov 03 09:21:32.452560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:32.452532 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0def2f1-9f7a-426a-b105-03e220428240" path="/var/lib/kubelet/pods/b0def2f1-9f7a-426a-b105-03e220428240/volumes" Nov 03 09:21:33.510897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:33.510844 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:35.560979 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:21:35.560942 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:21:39.053390 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:39.053301 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:44.612866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:44.612827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:44.756516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:44.756485 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:48.538486 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:48.538445 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:49.645062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:49.645028 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:50.077226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.077119 4543 generic.go:356] "Generic (PLEG): container finished" podID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerID="d68a4ff8f719418e8f2ac5f84538480e6889a7578089c10dec2d99e31f77ad70" exitCode=0 Nov 03 09:21:50.077226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.077182 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerDied","Data":"d68a4ff8f719418e8f2ac5f84538480e6889a7578089c10dec2d99e31f77ad70"} Nov 03 09:21:50.077691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.077676 4543 scope.go:122] "RemoveContainer" containerID="d68a4ff8f719418e8f2ac5f84538480e6889a7578089c10dec2d99e31f77ad70" Nov 03 09:21:50.079373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.079349 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:21:50.079441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.079413 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:21:50.079476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:50.079444 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:21:51.108612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:51.108578 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerStarted","Data":"0694b2ae756be08d04f4cc52ad890f6db10f3a7640cf3fa4fb26320ba865d2b3"} Nov 03 09:21:51.118113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:51.118092 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:21:51.119463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:51.119444 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:21:51.119513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:51.119488 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 09:21:51.311254 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:51.311221 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:21:52.117821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:52.117787 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:21:52.118256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:52.117838 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 09:21:53.117924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:53.117890 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:21:53.118434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:53.117943 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 09:21:54.118212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:54.118176 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:21:54.118618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:54.118232 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 09:21:54.642423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:54.642390 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:21:55.117617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:55.117579 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 03 09:21:55.117790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:55.117638 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 03 09:21:56.121779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:56.121750 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:21:56.122970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:56.122951 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:21:59.736885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:21:59.736853 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:01.759706 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:01.759668 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:05.178510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:05.178478 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:05.312433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:05.312398 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:05.584652 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:22:05.584556 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:22:06.249541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:06.249509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:10.331873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:10.331841 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:11.405976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:11.405923 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:12.290535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.290505 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:22:12.291090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291077 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.291169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291091 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.291169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291104 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.291169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291108 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.291274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291219 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.291274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.291226 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0def2f1-9f7a-426a-b105-03e220428240" containerName="istio-proxy" Nov 03 09:22:12.294676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.294659 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.349450 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.349423 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-2fc57e-openshift-default-dockercfg-z2xxk\"" Nov 03 09:22:12.377833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.377793 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.402683 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403222 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403253 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403299 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403331 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403360 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5lbd\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403388 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403410 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403462 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.404184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.403532 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.504780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.504745 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.504822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.504857 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.504922 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.504973 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505019 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5lbd\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505064 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505161 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505100 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505202 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505240 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505520 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505545 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505630 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.505783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.505760 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.507597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.507566 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.508037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.508018 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.564611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.564542 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.564752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.564677 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5lbd\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\") pod \"e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.608382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.608350 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:12.736824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:12.736552 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:13.042408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.042381 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:22:13.043277 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:22:13.043251 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f908cc_f045_4b7f_be81_d7f44ff148a9.slice/crio-7973f243f8d06b0c625435e90da6a00b2f3b8fb59f850804d0704da9ae779510 WatchSource:0}: Error finding container 7973f243f8d06b0c625435e90da6a00b2f3b8fb59f850804d0704da9ae779510: Status 404 returned error can't find the container with id 7973f243f8d06b0c625435e90da6a00b2f3b8fb59f850804d0704da9ae779510 Nov 03 09:22:13.045906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.045871 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:22:13.046002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.045949 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:22:13.046002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.045992 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:22:13.243424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.243390 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerStarted","Data":"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d"} Nov 03 09:22:13.243594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.243443 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerStarted","Data":"7973f243f8d06b0c625435e90da6a00b2f3b8fb59f850804d0704da9ae779510"} Nov 03 09:22:13.366287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.366239 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podStartSLOduration=1.366225806 podStartE2EDuration="1.366225806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:22:13.365555904 +0000 UTC m=+2557.033596643" watchObservedRunningTime="2026-11-03 09:22:13.366225806 +0000 UTC m=+2557.034266543" Nov 03 09:22:13.609326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:13.609291 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:14.613742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:14.613715 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:15.255388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:15.255357 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:15.256083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:15.256061 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:22:15.478702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:15.478633 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:16.116284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:16.116256 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:16.257729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:16.257695 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:16.454628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:16.454554 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:19.393690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:19.393654 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:19.583766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:19.581685 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:20.660286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.660246 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:22:20.664441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.664416 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.745849 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.745816 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppzt\" (UniqueName: \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.746028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.745895 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.746211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.746173 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.746335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.746213 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.768757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.768738 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9ed110-serving-cert\"" Nov 03 09:22:20.771908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.771889 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:22:20.847403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.847370 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.847403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.847407 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.847601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.847456 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppzt\" (UniqueName: \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.847601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.847502 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.847674 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:22:20.847620 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-serving-cert: secret "maas-api-e2e-shared-b-w0-9ed110-serving-cert" not found Nov 03 09:22:20.847704 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:22:20.847694 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls podName:3534807f-4b41-43ad-b1e8-818679a21968 nodeName:}" failed. No retries permitted until 2026-11-03 09:22:21.347675552 +0000 UTC m=+2565.015716285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls") pod "maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" (UID: "3534807f-4b41-43ad-b1e8-818679a21968") : secret "maas-api-e2e-shared-b-w0-9ed110-serving-cert" not found Nov 03 09:22:20.848105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.848087 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.850090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.850067 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:20.851405 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.851336 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:20.909004 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:20.908976 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppzt\" (UniqueName: \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:21.001435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.001353 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:21.135013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.134980 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:21.361662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.361635 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:21.364123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.364106 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:21.578810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.578780 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:21.956476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:21.956446 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:22:21.957291 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:22:21.957265 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3534807f_4b41_43ad_b1e8_818679a21968.slice/crio-a4b4f1a897a1e0e3baae2adc3335219af8523e1605fde36e3824925aeef9f82e WatchSource:0}: Error finding container a4b4f1a897a1e0e3baae2adc3335219af8523e1605fde36e3824925aeef9f82e: Status 404 returned error can't find the container with id a4b4f1a897a1e0e3baae2adc3335219af8523e1605fde36e3824925aeef9f82e Nov 03 09:22:22.309797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:22.309712 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" event={"ID":"3534807f-4b41-43ad-b1e8-818679a21968","Type":"ContainerStarted","Data":"a4b4f1a897a1e0e3baae2adc3335219af8523e1605fde36e3824925aeef9f82e"} Nov 03 09:22:23.317444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:23.317406 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" event={"ID":"3534807f-4b41-43ad-b1e8-818679a21968","Type":"ContainerStarted","Data":"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e"} Nov 03 09:22:23.317941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:23.317578 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:23.437409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:23.437359 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" podStartSLOduration=2.597459057 podStartE2EDuration="3.437345246s" totalImagesPullingTime="839.886189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:22:23.435421084 +0000 UTC m=+2567.103461822" watchObservedRunningTime="2026-11-03 09:22:23.437345246 +0000 UTC m=+2567.105385990" Nov 03 09:22:26.661710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:26.661645 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:28.767013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:28.766980 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:28.894781 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:28.894749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:29.329839 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:29.329810 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:22:32.087878 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:32.087849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:35.560987 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:22:35.560944 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:22:37.226817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:37.226786 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:41.257347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:41.257315 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:42.787338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:42.787307 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:45.012542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:45.012511 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:45.636371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:45.636335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:48.349202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:48.349090 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:48.877937 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:48.877893 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:53.452359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:53.452316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:22:56.736911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.736874 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:22:56.737395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.737079 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" containerID="cri-o://2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce" gracePeriod=30 Nov 03 09:22:56.787472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.787440 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:22:56.787665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.787646 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" containerID="cri-o://9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34" gracePeriod=30 Nov 03 09:22:56.790356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.790329 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:22:56.790551 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.790528 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" containerID="cri-o://5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea" gracePeriod=30 Nov 03 09:22:56.838205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.838173 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:22:56.838425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:56.838402 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" containerID="cri-o://44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195" gracePeriod=30 Nov 03 09:22:58.579339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:22:58.579308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:01.806257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:01.806222 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:54214->10.134.0.90:15021: read: connection reset by peer" start-of-body= Nov 03 09:23:01.806667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:01.806294 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": read tcp 10.134.0.2:54214->10.134.0.90:15021: read: connection reset by peer" Nov 03 09:23:01.856704 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:01.856673 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 09:23:01.856859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:01.856721 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 09:23:02.066259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.066229 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:23:02.132894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.132859 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.133055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.132967 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.133055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.133015 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.133191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.133072 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.133133 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.134841 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.134883 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.135057 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.135325 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\") pod \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\" (UID: \"63c9017c-31c8-47c8-8d59-b6a6cd9d8835\") " Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.136103 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.136614 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.136704 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.136832 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.136897 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.139532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.139263 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:23:02.142326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.142288 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "kube-api-access-7fh98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.142549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.142523 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.142852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.142652 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token" pod "63c9017c-31c8-47c8-8d59-b6a6cd9d8835" (UID: "63c9017c-31c8-47c8-8d59-b6a6cd9d8835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.179019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.178995 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:23:02.181407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.181390 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:23:02.194550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.194529 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236418 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236458 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236486 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236511 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236547 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236582 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236639 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236673 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236710 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236745 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236786 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236812 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236841 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.236873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236886 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236913 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236938 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.236980 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237065 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237108 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237220 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237271 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237313 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237397 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237431 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\") pod \"0d520045-3d44-4d89-bf73-571f2fdd05da\" (UID: \"0d520045-3d44-4d89-bf73-571f2fdd05da\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237514 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\") pod \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\" (UID: \"79b6cff4-be1b-47e3-abfb-0bb2448ad226\") " Nov 03 09:23:02.237701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.237556 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\") pod \"89ceb1e6-0f29-417f-af2d-f462e90e609b\" (UID: \"89ceb1e6-0f29-417f-af2d-f462e90e609b\") " Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238230 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238252 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238267 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fh98\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-kube-api-access-7fh98\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238282 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238298 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238301 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.238634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238313 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238373 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238393 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238428 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c9017c-31c8-47c8-8d59-b6a6cd9d8835-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.238634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238515 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.239383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238711 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.239383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238791 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.239383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.238924 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.239884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.239514 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.240771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.240741 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.240868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.240802 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.241231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241205 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.241592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241566 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.242182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241753 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.242182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241766 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.242182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241886 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:23:02.242182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.241915 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:23:02.242415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.242203 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.242672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.242611 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:23:02.244513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.244488 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.244618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.244525 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "kube-api-access-fmkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.245327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245303 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:23:02.245431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245325 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:23:02.245431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245389 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:23:02.245431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245400 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.245604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245451 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.245604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:23:02.245710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.245692 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl" pod "79b6cff4-be1b-47e3-abfb-0bb2448ad226" (UID: "79b6cff4-be1b-47e3-abfb-0bb2448ad226"). InnerVolumeSpecName "kube-api-access-7gzbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.246221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.246194 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g" pod "0d520045-3d44-4d89-bf73-571f2fdd05da" (UID: "0d520045-3d44-4d89-bf73-571f2fdd05da"). InnerVolumeSpecName "kube-api-access-k2t8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.247032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.247009 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token" pod "89ceb1e6-0f29-417f-af2d-f462e90e609b" (UID: "89ceb1e6-0f29-417f-af2d-f462e90e609b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340069 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340100 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340110 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340119 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340128 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340165 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzbl\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-kube-api-access-7gzbl\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340175 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2t8g\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-kube-api-access-k2t8g\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340187 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340195 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340205 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340213 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340222 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340230 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340238 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkbw\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-kube-api-access-fmkbw\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340246 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340254 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d520045-3d44-4d89-bf73-571f2fdd05da-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340262 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340271 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340279 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340286 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340293 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d520045-3d44-4d89-bf73-571f2fdd05da-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340301 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89ceb1e6-0f29-417f-af2d-f462e90e609b-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340309 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89ceb1e6-0f29-417f-af2d-f462e90e609b-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340318 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340328 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89ceb1e6-0f29-417f-af2d-f462e90e609b-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340336 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b6cff4-be1b-47e3-abfb-0bb2448ad226-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.340504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.340343 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d520045-3d44-4d89-bf73-571f2fdd05da-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:23:02.577618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.577580 4543 generic.go:356] "Generic (PLEG): container finished" podID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerID="5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea" exitCode=0 Nov 03 09:23:02.577810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.577637 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" Nov 03 09:23:02.577810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.577661 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" event={"ID":"0d520045-3d44-4d89-bf73-571f2fdd05da","Type":"ContainerDied","Data":"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea"} Nov 03 09:23:02.577810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.577748 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z" event={"ID":"0d520045-3d44-4d89-bf73-571f2fdd05da","Type":"ContainerDied","Data":"91ec9716c05f83467658b3b7ed1b964499a5e0a7f6b574438e796a594bb3fe87"} Nov 03 09:23:02.577810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.577767 4543 scope.go:122] "RemoveContainer" containerID="5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea" Nov 03 09:23:02.579221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.579198 4543 generic.go:356] "Generic (PLEG): container finished" podID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerID="2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce" exitCode=0 Nov 03 09:23:02.579350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.579254 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" Nov 03 09:23:02.579417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.579285 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" event={"ID":"63c9017c-31c8-47c8-8d59-b6a6cd9d8835","Type":"ContainerDied","Data":"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce"} Nov 03 09:23:02.579531 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.579511 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" event={"ID":"63c9017c-31c8-47c8-8d59-b6a6cd9d8835","Type":"ContainerDied","Data":"d92f3a05dc6f71ed17dc45a160d88bef994a066524cc816a7bb5c83e7c1788a3"} Nov 03 09:23:02.580958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.580934 4543 generic.go:356] "Generic (PLEG): container finished" podID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerID="9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34" exitCode=0 Nov 03 09:23:02.581087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.581034 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" event={"ID":"79b6cff4-be1b-47e3-abfb-0bb2448ad226","Type":"ContainerDied","Data":"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34"} Nov 03 09:23:02.581087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.581072 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" Nov 03 09:23:02.581286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.581177 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr" event={"ID":"79b6cff4-be1b-47e3-abfb-0bb2448ad226","Type":"ContainerDied","Data":"3db8f5ed318003234a1ca2b9afc42e3d54f718f8ff6fb68b3a11b50a377f72a7"} Nov 03 09:23:02.582611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.582590 4543 generic.go:356] "Generic (PLEG): container finished" podID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerID="44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195" exitCode=0 Nov 03 09:23:02.582768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.582651 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" Nov 03 09:23:02.582768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.582655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" event={"ID":"89ceb1e6-0f29-417f-af2d-f462e90e609b","Type":"ContainerDied","Data":"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195"} Nov 03 09:23:02.582768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.582766 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9" event={"ID":"89ceb1e6-0f29-417f-af2d-f462e90e609b","Type":"ContainerDied","Data":"e5d57d45d0a30c0aea5b8203c7362f60cc6469fec00c7905e2ed9dab92d5c447"} Nov 03 09:23:02.601565 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.601534 4543 scope.go:122] "RemoveContainer" containerID="5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea" Nov 03 09:23:02.602700 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:02.602668 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea\": container with ID starting with 5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea not found: ID does not exist" containerID="5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea" Nov 03 09:23:02.603057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.602704 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea"} err="failed to get container status \"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea\": rpc error: code = NotFound desc = could not find container \"5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea\": container with ID starting with 5814a815931a0a526500819a634b6569889bce2731621a49ea342715ff8db6ea not found: ID does not exist" Nov 03 09:23:02.603057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.602731 4543 scope.go:122] "RemoveContainer" containerID="2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce" Nov 03 09:23:02.613415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.613291 4543 scope.go:122] "RemoveContainer" containerID="2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce" Nov 03 09:23:02.613833 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:02.613801 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce\": container with ID starting with 2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce not found: ID does not exist" containerID="2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce" Nov 03 09:23:02.613944 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.613844 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce"} err="failed to get container status \"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce\": rpc error: code = NotFound desc = could not find container \"2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce\": container with ID starting with 2e2ab2ad26e38cd9a16424a05079abbba499e9cfe11ed32bca99d807e4f490ce not found: ID does not exist" Nov 03 09:23:02.613944 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.613868 4543 scope.go:122] "RemoveContainer" containerID="9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34" Nov 03 09:23:02.623351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.623331 4543 scope.go:122] "RemoveContainer" containerID="9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34" Nov 03 09:23:02.623650 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:02.623629 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34\": container with ID starting with 9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34 not found: ID does not exist" containerID="9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34" Nov 03 09:23:02.623717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.623659 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34"} err="failed to get container status \"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34\": rpc error: code = NotFound desc = could not find container \"9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34\": container with ID starting with 9133f2aa5820d4997c3f5f156c624d9d1d0654f38cf44f648d2ff74bb77dcf34 not found: ID does not exist" Nov 03 09:23:02.623717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.623678 4543 scope.go:122] "RemoveContainer" containerID="44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195" Nov 03 09:23:02.632983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.632963 4543 scope.go:122] "RemoveContainer" containerID="44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195" Nov 03 09:23:02.633332 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:02.633309 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195\": container with ID starting with 44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195 not found: ID does not exist" containerID="44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195" Nov 03 09:23:02.633421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.633335 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195"} err="failed to get container status \"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195\": rpc error: code = NotFound desc = could not find container \"44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195\": container with ID starting with 44ca4844e4e648c1fd687c283aa87bff06d1a1eae20dd52d1cb2d54da665e195 not found: ID does not exist" Nov 03 09:23:02.714229 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.714200 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:23:02.719539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.719510 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8"] Nov 03 09:23:02.833532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.833501 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:23:02.839460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.839436 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-w7j4z"] Nov 03 09:23:02.951879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.951846 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:23:02.957807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.957782 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lwsq9"] Nov 03 09:23:02.979246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.979217 4543 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:23:02.979363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:02.979271 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-9hjv8" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:23:03.070818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:03.070781 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:23:03.077272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:03.077245 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-6dhdr"] Nov 03 09:23:03.532811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:03.532782 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:04.452108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:04.452075 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" path="/var/lib/kubelet/pods/0d520045-3d44-4d89-bf73-571f2fdd05da/volumes" Nov 03 09:23:04.452612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:04.452597 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" path="/var/lib/kubelet/pods/63c9017c-31c8-47c8-8d59-b6a6cd9d8835/volumes" Nov 03 09:23:04.453059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:04.453046 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" path="/var/lib/kubelet/pods/79b6cff4-be1b-47e3-abfb-0bb2448ad226/volumes" Nov 03 09:23:04.453823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:04.453809 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" path="/var/lib/kubelet/pods/89ceb1e6-0f29-417f-af2d-f462e90e609b/volumes" Nov 03 09:23:05.560532 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:05.560498 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:23:06.592184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:06.592123 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:08.482484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:08.482452 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:10.028663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:10.028613 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:13.467274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:13.467242 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:18.468002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:18.467968 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:23.267036 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:23.266994 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:23.469694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:23.469658 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:28.183654 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.183622 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:23:28.184194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184179 4543 cpu_manager.go:414] "containerMap: removing container" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" Nov 03 09:23:28.184248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184196 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" Nov 03 09:23:28.184248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184233 4543 cpu_manager.go:414] "containerMap: removing container" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" Nov 03 09:23:28.184248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184238 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" Nov 03 09:23:28.184248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184245 4543 cpu_manager.go:414] "containerMap: removing container" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184250 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184258 4543 cpu_manager.go:414] "containerMap: removing container" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184263 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184356 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d520045-3d44-4d89-bf73-571f2fdd05da" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184366 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ceb1e6-0f29-417f-af2d-f462e90e609b" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184372 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b6cff4-be1b-47e3-abfb-0bb2448ad226" containerName="istio-proxy" Nov 03 09:23:28.184401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.184378 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c9017c-31c8-47c8-8d59-b6a6cd9d8835" containerName="istio-proxy" Nov 03 09:23:28.189283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.189258 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.245240 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.245209 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:23:28.249363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.249337 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkf2g\" (UniqueName: \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.249484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.249400 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.249555 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.249532 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.249669 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.249652 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.292316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.292284 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-2fc57e-serving-cert\"" Nov 03 09:23:28.350795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.350758 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkf2g\" (UniqueName: \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.350966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.350834 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.350966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.350912 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.351077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.350980 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.351150 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:28.351094 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-serving-cert: secret "maas-api-e2e-explicit-2fc57e-serving-cert" not found Nov 03 09:23:28.351217 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:28.351208 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls podName:29415179-5984-46ce-9817-c7ba2a0dcd87 nodeName:}" failed. No retries permitted until 2026-11-03 09:23:28.851184197 +0000 UTC m=+2632.519224929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls") pod "maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" (UID: "29415179-5984-46ce-9817-c7ba2a0dcd87") : secret "maas-api-e2e-explicit-2fc57e-serving-cert" not found Nov 03 09:23:28.351482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.351460 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.353456 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.353434 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.369681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.369659 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:28.409796 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.409765 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkf2g\" (UniqueName: \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.534868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.534791 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:28.663594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.663554 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:28.867996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.867970 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:28.870498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:28.870475 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") pod \"maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:29.103389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:29.103343 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:29.489080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:29.489050 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:23:29.490127 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:23:29.490103 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29415179_5984_46ce_9817_c7ba2a0dcd87.slice/crio-0619d54c0f4a8fb240595f2d39a00630e2bf11d3257137c30f942f27e9e87c97 WatchSource:0}: Error finding container 0619d54c0f4a8fb240595f2d39a00630e2bf11d3257137c30f942f27e9e87c97: Status 404 returned error can't find the container with id 0619d54c0f4a8fb240595f2d39a00630e2bf11d3257137c30f942f27e9e87c97 Nov 03 09:23:29.766629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:29.766558 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" event={"ID":"29415179-5984-46ce-9817-c7ba2a0dcd87","Type":"ContainerStarted","Data":"0619d54c0f4a8fb240595f2d39a00630e2bf11d3257137c30f942f27e9e87c97"} Nov 03 09:23:30.774325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:30.774290 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" event={"ID":"29415179-5984-46ce-9817-c7ba2a0dcd87","Type":"ContainerStarted","Data":"8a3776b2b5459097c2df7965d4e0ce81f73b32d9e812f530c7b690fd510a2e52"} Nov 03 09:23:30.774717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:30.774357 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:30.892542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:30.892488 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" podStartSLOduration=2.024446629 podStartE2EDuration="2.892474598s" totalImagesPullingTime="868.027969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:23:30.890548563 +0000 UTC m=+2634.558589300" watchObservedRunningTime="2026-11-03 09:23:30.892474598 +0000 UTC m=+2634.560515364" Nov 03 09:23:33.576436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:33.576398 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:35.561046 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:23:35.561007 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:23:36.785444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:36.785415 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:23:38.518312 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:38.518272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:38.681310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:38.681279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:38.805705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:38.805629 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:43.572798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:43.572768 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:45.359827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:45.359775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:45.959261 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:45.959231 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:46.844183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:46.844083 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:48.942079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:48.942047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:49.247460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:49.247381 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:53.937860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:53.937810 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:56.969056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.969025 4543 generic.go:356] "Generic (PLEG): container finished" podID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerID="19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d" exitCode=0 Nov 03 09:23:56.969513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.969113 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerDied","Data":"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d"} Nov 03 09:23:56.969559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.969548 4543 scope.go:122] "RemoveContainer" containerID="19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d" Nov 03 09:23:56.971061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.971034 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:23:56.971156 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.971120 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:23:56.971211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:56.971174 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:23:57.977333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:57.977300 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerStarted","Data":"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3"} Nov 03 09:23:58.609474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:58.609435 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:23:58.611063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:58.611027 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 09:23:58.611262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:58.611082 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 09:23:58.941289 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:58.941228 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:23:59.609016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:59.608970 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 09:23:59.609455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:23:59.609064 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 09:24:00.012169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:00.006749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:00.255609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:00.255578 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:24:00.608720 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:00.608681 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 09:24:00.608910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:00.608734 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 09:24:01.609184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:01.609129 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 09:24:01.609605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:01.609219 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 09:24:02.609098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:02.609058 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 03 09:24:02.609296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:02.609126 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 03 09:24:03.614091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:03.614059 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:24:03.615131 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:03.615113 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:24:03.998606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:03.998531 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:05.560309 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:24:05.560272 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:24:06.968638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:06.968596 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:09.076780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:09.076752 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:14.161792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:14.161755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:19.134598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:19.134560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:24.161407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:24.161372 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:29.164101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:29.164067 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:33.265925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:33.265892 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:34.230744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:34.230717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:35.561741 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:24:35.561699 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:24:36.811850 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:36.811822 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:24:36.876887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:36.876857 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:24:39.252023 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:39.251989 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:44.253094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:44.253048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:45.705338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:45.705281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:49.232096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:49.232056 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:49.638243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:49.638210 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:51.143634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:51.143604 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:54.452864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:54.452823 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:24:59.433504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:24:59.433472 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:04.384731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:04.384684 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:05.561227 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:25:05.561190 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:25:07.316595 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:07.316559 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:09.417870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:09.417830 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:13.441740 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:13.441682 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:14.511009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:14.510969 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:19.537347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:19.537261 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:24.541900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:24.541871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:28.422085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:28.422047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:29.653351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:29.653316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:34.633760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:34.633729 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:35.560353 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:25:35.560319 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:25:39.725941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:39.725891 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:44.828837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:44.828802 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:47.695574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:47.695543 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:25:47.695959 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:47.695740 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" containerID="cri-o://c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631" gracePeriod=30 Nov 03 09:25:49.803542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:49.803506 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:49.941347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:49.941316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:52.958298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:52.958250 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:25:53.055592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055515 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055620 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055664 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055701 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055781 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055845 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.055915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055893 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.056181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055936 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:53.056181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.055960 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\") pod \"8e730438-f249-4bec-b0d5-10a63c7cb36b\" (UID: \"8e730438-f249-4bec-b0d5-10a63c7cb36b\") " Nov 03 09:25:53.056181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056023 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:25:53.056339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056298 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:53.056626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056567 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.056626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056590 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e730438-f249-4bec-b0d5-10a63c7cb36b-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.056626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056605 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.056626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056603 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:53.056877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.056659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:53.057994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.057971 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:25:53.058616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.058599 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "kube-api-access-cq92f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:53.058711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.058612 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:25:53.058711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.058632 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy" pod "8e730438-f249-4bec-b0d5-10a63c7cb36b" (UID: "8e730438-f249-4bec-b0d5-10a63c7cb36b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:25:53.157890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157852 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq92f\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-kube-api-access-cq92f\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.157890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157882 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.157890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157895 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.158222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157908 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.158222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157920 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.158222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.157930 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e730438-f249-4bec-b0d5-10a63c7cb36b-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:25:53.720686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.720647 4543 generic.go:356] "Generic (PLEG): container finished" podID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerID="c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631" exitCode=0 Nov 03 09:25:53.720876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.720710 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" Nov 03 09:25:53.720876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.720744 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" event={"ID":"8e730438-f249-4bec-b0d5-10a63c7cb36b","Type":"ContainerDied","Data":"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631"} Nov 03 09:25:53.720876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.720804 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n" event={"ID":"8e730438-f249-4bec-b0d5-10a63c7cb36b","Type":"ContainerDied","Data":"0c4c3beaa1c665569d98a6e9ca143fb7e5b4292f09b70842b4ec4c94e17fedde"} Nov 03 09:25:53.720876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.720827 4543 scope.go:122] "RemoveContainer" containerID="c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631" Nov 03 09:25:53.731341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.731323 4543 scope.go:122] "RemoveContainer" containerID="c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631" Nov 03 09:25:53.731602 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:25:53.731585 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631\": container with ID starting with c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631 not found: ID does not exist" containerID="c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631" Nov 03 09:25:53.731651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.731613 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631"} err="failed to get container status \"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631\": rpc error: code = NotFound desc = could not find container \"c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631\": container with ID starting with c0e51bed96acf26d2f5e301057c7ac0be8b29f58f89b3895daa8bb4413af9631 not found: ID does not exist" Nov 03 09:25:53.891705 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.891666 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:25:53.941950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:53.941921 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-lqp9n"] Nov 03 09:25:54.451569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:54.451536 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" path="/var/lib/kubelet/pods/8e730438-f249-4bec-b0d5-10a63c7cb36b/volumes" Nov 03 09:25:54.791119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:54.791044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:56.021507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:56.021471 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:25:59.850702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:25:59.850669 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:04.828549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:04.828516 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:05.561863 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:26:05.561826 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:26:09.846724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:09.846694 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:14.820230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:14.820195 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:19.763114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:19.763077 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:23.798098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:23.798056 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:24.423844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:24.423814 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:24.558404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:24.558373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:24.741524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:24.741452 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:29.746805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:29.746769 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:34.757999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:34.757963 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:35.561015 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:26:35.560970 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:26:39.742494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:39.742458 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:42.419269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.419231 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:26:42.419659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.419486 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerName="discovery" containerID="cri-o://848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62" gracePeriod=30 Nov 03 09:26:42.702328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.702305 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:26:42.813864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.813832 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.813886 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.813922 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.813987 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.814039 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814290 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.814112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.814290 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.814192 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\") pod \"a189e9e5-976c-46ee-b2ed-19adc708f803\" (UID: \"a189e9e5-976c-46ee-b2ed-19adc708f803\") " Nov 03 09:26:42.815199 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.815130 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:26:42.816431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.816403 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:26:42.816562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.816491 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:26:42.816727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.816710 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:26:42.816933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.816917 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:26:42.817104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.817081 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "kube-api-access-248lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:26:42.817104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.817084 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts" pod "a189e9e5-976c-46ee-b2ed-19adc708f803" (UID: "a189e9e5-976c-46ee-b2ed-19adc708f803"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:26:42.916326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916291 4543 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-cacerts\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916328 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248lc\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-kube-api-access-248lc\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916341 4543 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-kubeconfig\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916356 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-ca-configmap\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916371 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916384 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a189e9e5-976c-46ee-b2ed-19adc708f803-istio-csr-dns-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:42.916540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:42.916399 4543 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a189e9e5-976c-46ee-b2ed-19adc708f803-local-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:26:43.013977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.013892 4543 generic.go:356] "Generic (PLEG): container finished" podID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerID="848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62" exitCode=0 Nov 03 09:26:43.013977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.013943 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" Nov 03 09:26:43.014257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.013972 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" event={"ID":"a189e9e5-976c-46ee-b2ed-19adc708f803","Type":"ContainerDied","Data":"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62"} Nov 03 09:26:43.014257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.014037 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95" event={"ID":"a189e9e5-976c-46ee-b2ed-19adc708f803","Type":"ContainerDied","Data":"dbb4f1ebbb1407454e3bae6dc854c2d0d3d4105849fb94e6110ae3bffdcc2ff0"} Nov 03 09:26:43.014257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.014061 4543 scope.go:122] "RemoveContainer" containerID="848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62" Nov 03 09:26:43.024604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.024587 4543 scope.go:122] "RemoveContainer" containerID="848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62" Nov 03 09:26:43.024876 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:26:43.024858 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62\": container with ID starting with 848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62 not found: ID does not exist" containerID="848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62" Nov 03 09:26:43.024929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.024885 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62"} err="failed to get container status \"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62\": rpc error: code = NotFound desc = could not find container \"848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62\": container with ID starting with 848dd78b2b3aebd74ed5ffb4dc2c563318b7b9aac8d88d60f2955e48e32e8c62 not found: ID does not exist" Nov 03 09:26:43.184838 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.184803 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:26:43.190930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:43.190907 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-blw95"] Nov 03 09:26:44.451339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:44.451298 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" path="/var/lib/kubelet/pods/a189e9e5-976c-46ee-b2ed-19adc708f803/volumes" Nov 03 09:26:44.821546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:44.821466 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:45.517728 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:45.517695 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:49.843269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:49.843192 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:50.274064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:50.274033 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:50.578330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:50.578248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:54.792296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:54.792261 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:26:59.786470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:26:59.786439 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:01.266447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:01.266404 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:01.453315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:01.453281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:04.732839 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:04.732794 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:05.561113 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:27:05.561070 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:27:06.080530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:06.080492 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:09.705244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:09.705207 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:14.653621 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:14.653563 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:19.620350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:19.620313 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:24.648331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:24.648295 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:27.217294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:27.217265 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:27.344194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:27.344103 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:28.853930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:28.853865 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:30.034218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:30.034184 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:35.075527 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:35.075488 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:35.211853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:35.211798 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:35.560620 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:27:35.560589 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:27:40.146133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:40.146070 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:43.998638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:43.998604 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:27:43.999191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:43.998794 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" containerID="cri-o://72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73" gracePeriod=30 Nov 03 09:27:45.132183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:45.132147 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:47.703463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:47.703422 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:27:47.703988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:47.703677 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" containerID="cri-o://bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c" gracePeriod=30 Nov 03 09:27:48.022250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:48.022171 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:27:48.022250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:48.022237 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:27:49.278342 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.278285 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:27:49.300293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300265 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300307 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300346 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300385 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300409 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300480 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300572 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300623 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.300716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300673 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\") pod \"57cf8c22-e780-454b-8762-f58becc82ede\" (UID: \"57cf8c22-e780-454b-8762-f58becc82ede\") " Nov 03 09:27:49.301180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300670 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:49.301180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300846 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:49.301180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300967 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:49.301180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.301038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:49.301180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.300957 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:27:49.302724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.302704 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:27:49.302811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.302776 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:49.303121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.303099 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "kube-api-access-bxskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:49.303230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.303125 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token" pod "57cf8c22-e780-454b-8762-f58becc82ede" (UID: "57cf8c22-e780-454b-8762-f58becc82ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:49.402848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402807 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.402848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402846 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402861 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57cf8c22-e780-454b-8762-f58becc82ede-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402873 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402888 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxskj\" (UniqueName: \"kubernetes.io/projected/57cf8c22-e780-454b-8762-f58becc82ede-kube-api-access-bxskj\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402902 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402915 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57cf8c22-e780-454b-8762-f58becc82ede-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402928 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.403112 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.402941 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57cf8c22-e780-454b-8762-f58becc82ede-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:49.406702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.406669 4543 generic.go:356] "Generic (PLEG): container finished" podID="57cf8c22-e780-454b-8762-f58becc82ede" containerID="72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73" exitCode=0 Nov 03 09:27:49.407406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.406719 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" Nov 03 09:27:49.407406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.406743 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" event={"ID":"57cf8c22-e780-454b-8762-f58becc82ede","Type":"ContainerDied","Data":"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73"} Nov 03 09:27:49.407406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.406834 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd" event={"ID":"57cf8c22-e780-454b-8762-f58becc82ede","Type":"ContainerDied","Data":"e347bbb3370e8c13db3e7cfebebbd5b307da2dc6404327eea3184e5f87f4fbae"} Nov 03 09:27:49.407406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.406874 4543 scope.go:122] "RemoveContainer" containerID="72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73" Nov 03 09:27:49.419798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.419778 4543 scope.go:122] "RemoveContainer" containerID="72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73" Nov 03 09:27:49.420180 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:27:49.420126 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73\": container with ID starting with 72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73 not found: ID does not exist" containerID="72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73" Nov 03 09:27:49.420285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.420191 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73"} err="failed to get container status \"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73\": rpc error: code = NotFound desc = could not find container \"72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73\": container with ID starting with 72fc1e07fc75ba9eb4ae9bd28981486dfba36e8ea40f75e56dd4c4076817ab73 not found: ID does not exist" Nov 03 09:27:49.563575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.563497 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:27:49.568614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:49.568587 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-fl6hd"] Nov 03 09:27:50.128192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:50.128161 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:50.451557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:50.451481 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57cf8c22-e780-454b-8762-f58becc82ede" path="/var/lib/kubelet/pods/57cf8c22-e780-454b-8762-f58becc82ede/volumes" Nov 03 09:27:51.402943 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:51.402910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:52.976015 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:52.975954 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:27:53.046566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046537 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046587 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046643 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046672 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046706 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046810 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046876 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.046996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046904 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.046996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046909 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.046996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.046949 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\") pod \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\" (UID: \"9e03bd68-c33c-44a3-9e77-7c4c32d7f179\") " Nov 03 09:27:53.047245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047075 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.047245 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047229 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:27:53.047358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047324 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.048027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047665 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.048027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047853 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.048027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047869 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.048027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047883 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.048027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.047898 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.049420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.049397 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:27:53.049420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.049402 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:27:53.049653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.049636 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:53.049746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.049731 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5" pod "9e03bd68-c33c-44a3-9e77-7c4c32d7f179" (UID: "9e03bd68-c33c-44a3-9e77-7c4c32d7f179"). InnerVolumeSpecName "kube-api-access-th5j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:27:53.148958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.148919 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.148958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.148957 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.149187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.148973 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.149187 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.148988 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th5j5\" (UniqueName: \"kubernetes.io/projected/9e03bd68-c33c-44a3-9e77-7c4c32d7f179-kube-api-access-th5j5\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:27:53.433207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.433174 4543 generic.go:356] "Generic (PLEG): container finished" podID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerID="bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c" exitCode=0 Nov 03 09:27:53.433385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.433224 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" Nov 03 09:27:53.433385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.433261 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" event={"ID":"9e03bd68-c33c-44a3-9e77-7c4c32d7f179","Type":"ContainerDied","Data":"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c"} Nov 03 09:27:53.433385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.433323 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t" event={"ID":"9e03bd68-c33c-44a3-9e77-7c4c32d7f179","Type":"ContainerDied","Data":"47cf31462a545ab261979d98a082f62cd8fd96e1db76ce3e3981444efec1d04e"} Nov 03 09:27:53.433385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.433347 4543 scope.go:122] "RemoveContainer" containerID="bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c" Nov 03 09:27:53.443835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.443813 4543 scope.go:122] "RemoveContainer" containerID="bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c" Nov 03 09:27:53.444114 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:27:53.444094 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c\": container with ID starting with bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c not found: ID does not exist" containerID="bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c" Nov 03 09:27:53.444220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.444125 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c"} err="failed to get container status \"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c\": rpc error: code = NotFound desc = could not find container \"bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c\": container with ID starting with bbd57fb4a8fce8070034027e61817163cc9c98fd095a340711b1504dc667737c not found: ID does not exist" Nov 03 09:27:53.607296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.607256 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:27:53.653791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:53.653750 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-vdp9t"] Nov 03 09:27:54.451184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:54.451129 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" path="/var/lib/kubelet/pods/9e03bd68-c33c-44a3-9e77-7c4c32d7f179/volumes" Nov 03 09:27:55.053171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:55.053116 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:59.857087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:59.857052 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:27:59.987359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:27:59.987321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:00.127847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:00.127818 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:05.096275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:05.096242 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:05.561787 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:28:05.561745 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:28:10.128102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:10.128071 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:15.080471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:15.080439 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:20.254568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:20.254478 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:25.275258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:25.275219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:29.508167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.508122 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:28:29.508575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.508380 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" podUID="29415179-5984-46ce-9817-c7ba2a0dcd87" containerName="maas-api" containerID="cri-o://8a3776b2b5459097c2df7965d4e0ce81f73b32d9e812f530c7b690fd510a2e52" gracePeriod=30 Nov 03 09:28:29.597536 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.597499 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:29.637758 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.637726 4543 generic.go:356] "Generic (PLEG): container finished" podID="29415179-5984-46ce-9817-c7ba2a0dcd87" containerID="8a3776b2b5459097c2df7965d4e0ce81f73b32d9e812f530c7b690fd510a2e52" exitCode=0 Nov 03 09:28:29.637911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.637798 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" event={"ID":"29415179-5984-46ce-9817-c7ba2a0dcd87","Type":"ContainerDied","Data":"8a3776b2b5459097c2df7965d4e0ce81f73b32d9e812f530c7b690fd510a2e52"} Nov 03 09:28:29.777985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.777961 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:28:29.872798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.872764 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") pod \"29415179-5984-46ce-9817-c7ba2a0dcd87\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " Nov 03 09:28:29.872968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.872830 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\") pod \"29415179-5984-46ce-9817-c7ba2a0dcd87\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " Nov 03 09:28:29.872968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.872900 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\" (UniqueName: \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\") pod \"29415179-5984-46ce-9817-c7ba2a0dcd87\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " Nov 03 09:28:29.873042 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.873009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\") pod \"29415179-5984-46ce-9817-c7ba2a0dcd87\" (UID: \"29415179-5984-46ce-9817-c7ba2a0dcd87\") " Nov 03 09:28:29.873218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.873190 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca" pod "29415179-5984-46ce-9817-c7ba2a0dcd87" (UID: "29415179-5984-46ce-9817-c7ba2a0dcd87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:28:29.873526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.873504 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29415179-5984-46ce-9817-c7ba2a0dcd87-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:29.875177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.875152 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls" pod "29415179-5984-46ce-9817-c7ba2a0dcd87" (UID: "29415179-5984-46ce-9817-c7ba2a0dcd87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:28:29.875295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.875273 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs" pod "29415179-5984-46ce-9817-c7ba2a0dcd87" (UID: "29415179-5984-46ce-9817-c7ba2a0dcd87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:28:29.875360 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.875323 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g" pod "29415179-5984-46ce-9817-c7ba2a0dcd87" (UID: "29415179-5984-46ce-9817-c7ba2a0dcd87"). InnerVolumeSpecName "kube-api-access-dkf2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:28:29.974619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.974582 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:29.974619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.974612 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkf2g\" (UniqueName: \"kubernetes.io/projected/29415179-5984-46ce-9817-c7ba2a0dcd87-kube-api-access-dkf2g\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:29.974619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:29.974623 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29415179-5984-46ce-9817-c7ba2a0dcd87-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:30.644988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:30.644960 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" Nov 03 09:28:30.645512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:30.644961 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp" event={"ID":"29415179-5984-46ce-9817-c7ba2a0dcd87","Type":"ContainerDied","Data":"0619d54c0f4a8fb240595f2d39a00630e2bf11d3257137c30f942f27e9e87c97"} Nov 03 09:28:30.645512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:30.645088 4543 scope.go:122] "RemoveContainer" containerID="8a3776b2b5459097c2df7965d4e0ce81f73b32d9e812f530c7b690fd510a2e52" Nov 03 09:28:30.886549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:30.886514 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:28:30.889028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:30.889006 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2fc57e-6df855ccb-wrbvp"] Nov 03 09:28:31.392251 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:31.392217 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:31.521275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:31.521237 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:31.964357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:31.964323 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:28:31.964872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:31.964598 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" containerID="cri-o://2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3" gracePeriod=30 Nov 03 09:28:31.997401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:31.997343 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:32.450861 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:32.450817 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29415179-5984-46ce-9817-c7ba2a0dcd87" path="/var/lib/kubelet/pods/29415179-5984-46ce-9817-c7ba2a0dcd87/volumes" Nov 03 09:28:32.584930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:32.584893 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:33.617475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:33.617431 4543 patch_prober.go:29] interesting pod/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:28:33.617860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:33.617489 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:28:35.245775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.245739 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:28:35.246525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246504 4543 cpu_manager.go:414] "containerMap: removing container" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246528 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246555 4543 cpu_manager.go:414] "containerMap: removing container" podUID="29415179-5984-46ce-9817-c7ba2a0dcd87" containerName="maas-api" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246563 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29415179-5984-46ce-9817-c7ba2a0dcd87" containerName="maas-api" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246573 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246580 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246608 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerName="discovery" Nov 03 09:28:35.246615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246615 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerName="discovery" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246635 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246643 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246798 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="57cf8c22-e780-454b-8762-f58becc82ede" containerName="istio-proxy" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246813 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="29415179-5984-46ce-9817-c7ba2a0dcd87" containerName="maas-api" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246822 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e03bd68-c33c-44a3-9e77-7c4c32d7f179" containerName="istio-proxy" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246834 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a189e9e5-976c-46ee-b2ed-19adc708f803" containerName="discovery" Nov 03 09:28:35.246952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.246845 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e730438-f249-4bec-b0d5-10a63c7cb36b" containerName="istio-proxy" Nov 03 09:28:35.251605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.251583 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.306638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.306606 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4ba21a-openshift-default-dockercfg-jldrq\"" Nov 03 09:28:35.341607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.341568 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:35.349367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349340 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349404 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349470 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349496 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349541 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349517 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349655 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349757 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcxf\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349838 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.349886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.349879 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.359273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.359250 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:28:35.451761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.451726 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcxf\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.451970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.451810 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.451970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.451844 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.451970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.451892 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452046 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452100 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452128 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452190 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452254 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452290 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452311 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452505 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452534 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.452912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.452893 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.454523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.454499 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.455004 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.454980 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.512154 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.512055 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcxf\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.512400 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.512379 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\") pod \"e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.560791 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:28:35.560755 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:28:35.565104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.565078 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:35.997681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:35.997649 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:28:36.036834 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:28:36.036803 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e07230f_7c88_4e13_bc2b_7b11058dd44e.slice/crio-c24fb386735acf07fe0cca92e01efa0ed691f83fb198be069e73233bbf5c3b64 WatchSource:0}: Error finding container c24fb386735acf07fe0cca92e01efa0ed691f83fb198be069e73233bbf5c3b64: Status 404 returned error can't find the container with id c24fb386735acf07fe0cca92e01efa0ed691f83fb198be069e73233bbf5c3b64 Nov 03 09:28:36.039579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.039548 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:28:36.039663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.039616 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:28:36.039663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.039642 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:28:36.499084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.499042 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:36.685003 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.684965 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" event={"ID":"5e07230f-7c88-4e13-bc2b-7b11058dd44e","Type":"ContainerStarted","Data":"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674"} Nov 03 09:28:36.685205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.685051 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" event={"ID":"5e07230f-7c88-4e13-bc2b-7b11058dd44e","Type":"ContainerStarted","Data":"c24fb386735acf07fe0cca92e01efa0ed691f83fb198be069e73233bbf5c3b64"} Nov 03 09:28:36.787430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.787349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:36.806101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:36.806051 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" podStartSLOduration=1.806033303 podStartE2EDuration="1.806033303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:28:36.804080717 +0000 UTC m=+2940.472121461" watchObservedRunningTime="2026-11-03 09:28:36.806033303 +0000 UTC m=+2940.474074044" Nov 03 09:28:37.183259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.183216 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:37.225403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.225374 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:28:37.323908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.323845 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:37.375911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.375881 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.375929 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.375957 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376061 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376127 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376174 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376325 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376364 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376394 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\") pod \"93f908cc-f045-4b7f-be81-d7f44ff148a9\" (UID: \"93f908cc-f045-4b7f-be81-d7f44ff148a9\") " Nov 03 09:28:37.376498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376395 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:37.376498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376438 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:37.376873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.376822 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:28:37.377286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377249 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:37.377286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377265 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:37.377928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377902 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.378044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377933 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f908cc-f045-4b7f-be81-d7f44ff148a9-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.378044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377950 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.378044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377963 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.378044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.377977 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.378584 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.378559 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:28:37.378861 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.378841 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "kube-api-access-l5lbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:28:37.379096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.379073 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:28:37.379334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.379313 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy" pod "93f908cc-f045-4b7f-be81-d7f44ff148a9" (UID: "93f908cc-f045-4b7f-be81-d7f44ff148a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:28:37.463215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.463180 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:37.479108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.479082 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.479108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.479106 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.479108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.479115 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.479364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.479124 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5lbd\" (UniqueName: \"kubernetes.io/projected/93f908cc-f045-4b7f-be81-d7f44ff148a9-kube-api-access-l5lbd\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:28:37.565340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.565306 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:37.571406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.571382 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:37.692301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692211 4543 generic.go:356] "Generic (PLEG): container finished" podID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerID="2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3" exitCode=0 Nov 03 09:28:37.692301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692261 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" Nov 03 09:28:37.692525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692253 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerDied","Data":"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3"} Nov 03 09:28:37.692525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692383 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb" event={"ID":"93f908cc-f045-4b7f-be81-d7f44ff148a9","Type":"ContainerDied","Data":"7973f243f8d06b0c625435e90da6a00b2f3b8fb59f850804d0704da9ae779510"} Nov 03 09:28:37.692525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692407 4543 scope.go:122] "RemoveContainer" containerID="2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3" Nov 03 09:28:37.692744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.692728 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:37.694376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.694363 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:28:37.703615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.703595 4543 scope.go:122] "RemoveContainer" containerID="19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d" Nov 03 09:28:37.712504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.712488 4543 scope.go:122] "RemoveContainer" containerID="2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3" Nov 03 09:28:37.712781 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:28:37.712761 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3\": container with ID starting with 2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3 not found: ID does not exist" containerID="2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3" Nov 03 09:28:37.712905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.712793 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3"} err="failed to get container status \"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3\": rpc error: code = NotFound desc = could not find container \"2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3\": container with ID starting with 2fdf37636ebd79c3de5e2d0fb5f2766b41dacc2bce847027eee2e7f658e2f4c3 not found: ID does not exist" Nov 03 09:28:37.712905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.712823 4543 scope.go:122] "RemoveContainer" containerID="19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d" Nov 03 09:28:37.713090 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:28:37.713071 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d\": container with ID starting with 19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d not found: ID does not exist" containerID="19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d" Nov 03 09:28:37.713129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.713097 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d"} err="failed to get container status \"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d\": rpc error: code = NotFound desc = could not find container \"19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d\": container with ID starting with 19e47503263bc385575aba52a3b23a76bad124f4a7c27ee4f4bb876b7348c92d not found: ID does not exist" Nov 03 09:28:37.990461 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.990383 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:28:37.994323 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:37.994297 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-2fc57e-openshift-default-5584f6b849-f4tlb"] Nov 03 09:28:38.453281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:38.453250 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" path="/var/lib/kubelet/pods/93f908cc-f045-4b7f-be81-d7f44ff148a9/volumes" Nov 03 09:28:38.980430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:38.980393 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:39.132287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:39.132254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:40.756290 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.756255 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:28:40.756818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.756801 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:28:40.756863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.756822 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:28:40.756956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.756947 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:28:40.761983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.761960 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.829801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.829753 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:28:40.845511 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.845482 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:40.864777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.864748 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4ba21a\"/\"kube-root-ca.crt\"" Nov 03 09:28:40.864777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.864767 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4ba21a\"/\"noten-model-4ba21a-kserve-self-signed-certs\"" Nov 03 09:28:40.868472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.868449 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4ba21a\"/\"openshift-service-ca.crt\"" Nov 03 09:28:40.868586 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.868501 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4ba21a\"/\"default-dockercfg-l5xkm\"" Nov 03 09:28:40.921349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921304 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.921349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921357 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9vf\" (UniqueName: \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.921594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921565 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.921666 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921652 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.921713 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921679 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:40.921756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:40.921718 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022472 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022631 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022728 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022769 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9vf\" (UniqueName: \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022926 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022866 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.022926 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.022911 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.023371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.023351 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.025062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.025038 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.025323 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.025309 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.082854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.082826 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9vf\" (UniqueName: \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\") pod \"noten-model-4ba21a-kserve-67546bb7b5-vflnx\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.374105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.374078 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:41.752325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.752288 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:28:41.753599 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:28:41.753575 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc33d557d_ee40_4623_a7c1_a5267251bb9d.slice/crio-b9988007c7d08b87439675d974489b6317770c29bc9da5bc217121b68a0af9de WatchSource:0}: Error finding container b9988007c7d08b87439675d974489b6317770c29bc9da5bc217121b68a0af9de: Status 404 returned error can't find the container with id b9988007c7d08b87439675d974489b6317770c29bc9da5bc217121b68a0af9de Nov 03 09:28:41.874817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:41.874782 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:42.723933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:42.723897 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" event={"ID":"c33d557d-ee40-4623-a7c1-a5267251bb9d","Type":"ContainerStarted","Data":"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649"} Nov 03 09:28:42.724109 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:42.723950 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" event={"ID":"c33d557d-ee40-4623-a7c1-a5267251bb9d","Type":"ContainerStarted","Data":"b9988007c7d08b87439675d974489b6317770c29bc9da5bc217121b68a0af9de"} Nov 03 09:28:42.846105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:42.846044 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" podStartSLOduration=2.846027508 podStartE2EDuration="2.846027508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:28:42.844414548 +0000 UTC m=+2946.512455289" watchObservedRunningTime="2026-11-03 09:28:42.846027508 +0000 UTC m=+2946.514068246" Nov 03 09:28:43.374847 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:43.374815 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:46.888577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:46.888544 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:51.374799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:51.374764 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:51.387870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:51.387844 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:51.749642 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:51.749561 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:51.785554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:51.785528 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:28:51.886790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:51.886753 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:55.634909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:55.634874 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:28:56.838351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:28:56.838320 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:01.834122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:01.834089 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:05.561872 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:29:05.561833 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:29:06.839824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:06.839787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:11.952992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:11.952960 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:16.948633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:16.948590 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:19.734984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:19.734947 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:29:19.735580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:19.735185 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerName="istio-proxy" containerID="cri-o://eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674" gracePeriod=30 Nov 03 09:29:19.739911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:19.739871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:20.028859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:20.028786 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:20.168884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:20.168849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:21.942890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:21.942860 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:22.695346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:22.695308 4543 patch_prober.go:29] interesting pod/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:29:22.695517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:22.695360 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:29:24.899750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.899719 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:29:24.964563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.964523 4543 generic.go:356] "Generic (PLEG): container finished" podID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerID="eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674" exitCode=0 Nov 03 09:29:24.964744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.964579 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" Nov 03 09:29:24.964744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.964604 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" event={"ID":"5e07230f-7c88-4e13-bc2b-7b11058dd44e","Type":"ContainerDied","Data":"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674"} Nov 03 09:29:24.964744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.964663 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2" event={"ID":"5e07230f-7c88-4e13-bc2b-7b11058dd44e","Type":"ContainerDied","Data":"c24fb386735acf07fe0cca92e01efa0ed691f83fb198be069e73233bbf5c3b64"} Nov 03 09:29:24.964744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.964688 4543 scope.go:122] "RemoveContainer" containerID="eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674" Nov 03 09:29:24.974118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.974101 4543 scope.go:122] "RemoveContainer" containerID="eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674" Nov 03 09:29:24.974404 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:29:24.974386 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674\": container with ID starting with eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674 not found: ID does not exist" containerID="eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674" Nov 03 09:29:24.974484 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:24.974412 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674"} err="failed to get container status \"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674\": rpc error: code = NotFound desc = could not find container \"eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674\": container with ID starting with eb432608a46d92ac67df96c60b1de12024b8d6a2cc5044c43d4eb58905f8c674 not found: ID does not exist" Nov 03 09:29:25.066063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.065987 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066079 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066110 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066177 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066301 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066435 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066436 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:25.066539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066462 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:29:25.066539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066491 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066532 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066569 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\") pod \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\" (UID: \"5e07230f-7c88-4e13-bc2b-7b11058dd44e\") " Nov 03 09:29:25.066801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066574 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:25.066801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066588 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:25.066965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.066814 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:25.067330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.067265 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.067330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.067291 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.067330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.067307 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.067596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.067354 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.067596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.067369 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.068607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.068582 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "kube-api-access-tqcxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:25.068767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.068746 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:29:25.068819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.068808 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:25.068922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.068897 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy" pod "5e07230f-7c88-4e13-bc2b-7b11058dd44e" (UID: "5e07230f-7c88-4e13-bc2b-7b11058dd44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:25.136972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.136938 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:29:25.168668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.168638 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.168668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.168666 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqcxf\" (UniqueName: \"kubernetes.io/projected/5e07230f-7c88-4e13-bc2b-7b11058dd44e-kube-api-access-tqcxf\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.168668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.168677 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.168896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.168686 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e07230f-7c88-4e13-bc2b-7b11058dd44e-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:25.229080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.229048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:25.389913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.389879 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:29:25.395914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:25.395886 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4ba21a-openshift-default-66c49584cf-tm9r2"] Nov 03 09:29:26.451203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:26.451172 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" path="/var/lib/kubelet/pods/5e07230f-7c88-4e13-bc2b-7b11058dd44e/volumes" Nov 03 09:29:26.952026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:26.951996 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:30.644553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:30.644518 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:32.066464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:32.066429 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:35.560845 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:29:35.560802 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:29:36.848697 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:36.848671 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:29:36.914821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:36.914794 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:29:37.194696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:37.194663 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:40.158250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.158209 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" podUID="c33d557d-ee40-4623-a7c1-a5267251bb9d" containerName="main" containerID="cri-o://dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649" gracePeriod=45 Nov 03 09:29:40.414357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.414296 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:29:40.444864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.444831 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.444885 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.444948 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.444978 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\" (UniqueName: \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445023 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445077 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\") pod \"c33d557d-ee40-4623-a7c1-a5267251bb9d\" (UID: \"c33d557d-ee40-4623-a7c1-a5267251bb9d\") " Nov 03 09:29:40.445329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445229 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:40.445329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445246 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:40.445477 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445462 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:40.445649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445631 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:40.445718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445656 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:40.445718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.445669 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:40.447382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.447355 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:29:40.447867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.447843 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "kube-api-access-nx9vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:29:40.447949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.447915 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm" pod "c33d557d-ee40-4623-a7c1-a5267251bb9d" (UID: "c33d557d-ee40-4623-a7c1-a5267251bb9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:29:40.547295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.547224 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9vf\" (UniqueName: \"kubernetes.io/projected/c33d557d-ee40-4623-a7c1-a5267251bb9d-kube-api-access-nx9vf\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:40.547295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.547255 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c33d557d-ee40-4623-a7c1-a5267251bb9d-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:40.547295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:40.547269 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33d557d-ee40-4623-a7c1-a5267251bb9d-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:29:41.062311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.062278 4543 generic.go:356] "Generic (PLEG): container finished" podID="c33d557d-ee40-4623-a7c1-a5267251bb9d" containerID="dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649" exitCode=0 Nov 03 09:29:41.062496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.062317 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" Nov 03 09:29:41.062496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.062367 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" event={"ID":"c33d557d-ee40-4623-a7c1-a5267251bb9d","Type":"ContainerDied","Data":"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649"} Nov 03 09:29:41.062496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.062414 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx" event={"ID":"c33d557d-ee40-4623-a7c1-a5267251bb9d","Type":"ContainerDied","Data":"b9988007c7d08b87439675d974489b6317770c29bc9da5bc217121b68a0af9de"} Nov 03 09:29:41.062496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.062435 4543 scope.go:122] "RemoveContainer" containerID="dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649" Nov 03 09:29:41.072256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.072232 4543 scope.go:122] "RemoveContainer" containerID="dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649" Nov 03 09:29:41.072517 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:29:41.072493 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649\": container with ID starting with dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649 not found: ID does not exist" containerID="dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649" Nov 03 09:29:41.072589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.072530 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649"} err="failed to get container status \"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649\": rpc error: code = NotFound desc = could not find container \"dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649\": container with ID starting with dcf81eaba4e67b56a0cc870cb7bda4307ecc2230c17c18ce9936bd2a55485649 not found: ID does not exist" Nov 03 09:29:41.232258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.232228 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:29:41.236970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:41.236947 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4ba21a/noten-model-4ba21a-kserve-67546bb7b5-vflnx"] Nov 03 09:29:42.140925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:42.140887 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:42.451638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:42.451563 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33d557d-ee40-4623-a7c1-a5267251bb9d" path="/var/lib/kubelet/pods/c33d557d-ee40-4623-a7c1-a5267251bb9d/volumes" Nov 03 09:29:47.141771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:47.141694 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:52.102808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:52.102775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:52.235994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:52.235963 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:53.731371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:53.731332 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:54.532366 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:54.532333 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:55.977893 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:55.977837 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:57.100715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:57.100678 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:58.865017 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:58.864983 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:29:59.834204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:29:59.834166 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:00.260436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.260404 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:00.260957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.260941 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c33d557d-ee40-4623-a7c1-a5267251bb9d" containerName="main" Nov 03 09:30:00.260957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.260957 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33d557d-ee40-4623-a7c1-a5267251bb9d" containerName="main" Nov 03 09:30:00.261115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.260985 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerName="istio-proxy" Nov 03 09:30:00.261115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.260993 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerName="istio-proxy" Nov 03 09:30:00.261115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.261011 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:30:00.261115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.261017 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:30:00.261115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.261103 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e07230f-7c88-4e13-bc2b-7b11058dd44e" containerName="istio-proxy" Nov 03 09:30:00.261393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.261119 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f908cc-f045-4b7f-be81-d7f44ff148a9" containerName="istio-proxy" Nov 03 09:30:00.261393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.261154 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33d557d-ee40-4623-a7c1-a5267251bb9d" containerName="main" Nov 03 09:30:00.265978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.265960 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.292224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.292196 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:00.296680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.296653 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.325336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.325307 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:00.362457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.362421 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:00.372031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.372001 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:30:00.373633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.373589 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.373794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.373681 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqlm\" (UniqueName: \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.474459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.474429 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqlm\" (UniqueName: \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.474642 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.474479 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4h9\" (UniqueName: \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.474642 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.474617 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.474747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.474651 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.475210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.475187 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.534446 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.534380 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqlm\" (UniqueName: \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.576062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.576027 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.576238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.576110 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4h9\" (UniqueName: \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.576740 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.576721 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.580761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.580740 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:00.663338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.659305 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4h9\" (UniqueName: \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.917100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.917073 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:00.998815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:00.998788 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:01.001388 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:01.001355 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb786bf7d_7b2e_41d5_a6ae_4fcc66a18dca.slice/crio-3a8a83a5d175f876ad38c3975f36cf448c143f95a48c4a315dc36fd459b650ab WatchSource:0}: Error finding container 3a8a83a5d175f876ad38c3975f36cf448c143f95a48c4a315dc36fd459b650ab: Status 404 returned error can't find the container with id 3a8a83a5d175f876ad38c3975f36cf448c143f95a48c4a315dc36fd459b650ab Nov 03 09:30:01.003648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:01.003625 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:30:01.182271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:01.182194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:01.185572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:01.185543 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerStarted","Data":"3a8a83a5d175f876ad38c3975f36cf448c143f95a48c4a315dc36fd459b650ab"} Nov 03 09:30:01.296063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:01.296031 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:01.297030 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:01.297004 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2662c97_5fb7_4468_bc36_8a1b8367eb50.slice/crio-e9bc76bf3d3877a3bc0187f28d44a6e3773ef25ce13dd661bbfa96bccb8ed6c3 WatchSource:0}: Error finding container e9bc76bf3d3877a3bc0187f28d44a6e3773ef25ce13dd661bbfa96bccb8ed6c3: Status 404 returned error can't find the container with id e9bc76bf3d3877a3bc0187f28d44a6e3773ef25ce13dd661bbfa96bccb8ed6c3 Nov 03 09:30:02.143620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.143587 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:02.193394 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.193362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerStarted","Data":"e9bc76bf3d3877a3bc0187f28d44a6e3773ef25ce13dd661bbfa96bccb8ed6c3"} Nov 03 09:30:02.285403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.285374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:02.377299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.377265 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-a57494"] Nov 03 09:30:02.382901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.382878 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:02.435540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.435447 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-a57494"] Nov 03 09:30:02.482287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.482239 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-77bkh\"" Nov 03 09:30:02.498603 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.498575 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7sc\" (UniqueName: \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\") pod \"test-curl-7381-a57494\" (UID: \"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6\") " pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:02.599983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.599945 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7sc\" (UniqueName: \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\") pod \"test-curl-7381-a57494\" (UID: \"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6\") " pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:02.662653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.662625 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7sc\" (UniqueName: \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\") pod \"test-curl-7381-a57494\" (UID: \"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6\") " pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:02.695823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.695752 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:02.873211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:02.873180 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:03.019356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.019277 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:03.071285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.071254 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-a57494"] Nov 03 09:30:03.086586 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:03.086493 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ec5912_3ed0_40ef_a3c9_e9d30eccbae6.slice/crio-3bf3faa9b61405368324b2e80244215ca4c6a1bf9149f3799313e8b508d4039e WatchSource:0}: Error finding container 3bf3faa9b61405368324b2e80244215ca4c6a1bf9149f3799313e8b508d4039e: Status 404 returned error can't find the container with id 3bf3faa9b61405368324b2e80244215ca4c6a1bf9149f3799313e8b508d4039e Nov 03 09:30:03.092986 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.092961 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7387-21acf6"] Nov 03 09:30:03.099171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.099126 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:03.152669 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.152636 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-21acf6"] Nov 03 09:30:03.201763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.201730 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-a57494" event={"ID":"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6","Type":"ContainerStarted","Data":"3bf3faa9b61405368324b2e80244215ca4c6a1bf9149f3799313e8b508d4039e"} Nov 03 09:30:03.206201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.206178 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7855t\" (UniqueName: \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\") pod \"test-curl-7387-21acf6\" (UID: \"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2\") " pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:03.307799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.307714 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7855t\" (UniqueName: \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\") pod \"test-curl-7387-21acf6\" (UID: \"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2\") " pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:03.366722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.366694 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7855t\" (UniqueName: \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\") pod \"test-curl-7387-21acf6\" (UID: \"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2\") " pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:03.412299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.412267 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:03.788942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:03.788913 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-21acf6"] Nov 03 09:30:03.790564 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:03.790533 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5da36a7_fa55_4cd1_829c_a4e1f9c70ed2.slice/crio-20565d5b018b508361c8cf47f53ca63448a0bde35150d0f84226ce433d5be4b4 WatchSource:0}: Error finding container 20565d5b018b508361c8cf47f53ca63448a0bde35150d0f84226ce433d5be4b4: Status 404 returned error can't find the container with id 20565d5b018b508361c8cf47f53ca63448a0bde35150d0f84226ce433d5be4b4 Nov 03 09:30:04.212049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:04.212017 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-21acf6" event={"ID":"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2","Type":"ContainerStarted","Data":"20565d5b018b508361c8cf47f53ca63448a0bde35150d0f84226ce433d5be4b4"} Nov 03 09:30:05.222947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:05.222886 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-21acf6" event={"ID":"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2","Type":"ContainerStarted","Data":"fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006"} Nov 03 09:30:05.225231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:05.225154 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-a57494" event={"ID":"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6","Type":"ContainerStarted","Data":"40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50"} Nov 03 09:30:05.465997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:05.465938 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-a57494" podStartSLOduration=2.099549171 podStartE2EDuration="3.465919132s" totalImagesPullingTime="1.366369961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:05.463698793 +0000 UTC m=+3029.131739532" watchObservedRunningTime="2026-11-03 09:30:05.465919132 +0000 UTC m=+3029.133959875" Nov 03 09:30:05.466445 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:05.466413 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7387-21acf6" podStartSLOduration=1.7544228830000002 podStartE2EDuration="2.466402682s" totalImagesPullingTime="711.979799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:05.344289603 +0000 UTC m=+3029.012330342" watchObservedRunningTime="2026-11-03 09:30:05.466402682 +0000 UTC m=+3029.134443427" Nov 03 09:30:05.573814 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:05.573731 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:30:06.232678 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:06.232641 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerStarted","Data":"6c0d1022611892b0f9338bf4661290ed1d1a7f0041a0c042b229bc211a85c9e0"} Nov 03 09:30:06.351690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:06.351601 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" podStartSLOduration=1.969493309 podStartE2EDuration="6.351586274s" totalImagesPullingTime="4.382092965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:06.349365522 +0000 UTC m=+3030.017406262" watchObservedRunningTime="2026-11-03 09:30:06.351586274 +0000 UTC m=+3030.019627013" Nov 03 09:30:06.770595 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:06.770562 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7387-21acf6"] Nov 03 09:30:06.777326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:06.777289 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7387-21acf6"] Nov 03 09:30:07.026169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.026056 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-a57494"] Nov 03 09:30:07.032339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.032306 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-a57494"] Nov 03 09:30:07.208651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.208615 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:07.239877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.239845 4543 generic.go:356] "Generic (PLEG): container finished" podID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerID="6c0d1022611892b0f9338bf4661290ed1d1a7f0041a0c042b229bc211a85c9e0" exitCode=60 Nov 03 09:30:07.240340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.239935 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerDied","Data":"6c0d1022611892b0f9338bf4661290ed1d1a7f0041a0c042b229bc211a85c9e0"} Nov 03 09:30:07.240340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.240200 4543 scope.go:122] "RemoveContainer" containerID="6c0d1022611892b0f9338bf4661290ed1d1a7f0041a0c042b229bc211a85c9e0" Nov 03 09:30:07.241635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.241614 4543 generic.go:356] "Generic (PLEG): container finished" podID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerID="46678fd61a822f45d1f654b5fdfea37e06ad7bbe9e908b1621d44ad9bf7592e6" exitCode=60 Nov 03 09:30:07.241754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.241691 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerDied","Data":"46678fd61a822f45d1f654b5fdfea37e06ad7bbe9e908b1621d44ad9bf7592e6"} Nov 03 09:30:07.241859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.241839 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-a57494" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" containerName="test-curl-7381-a57494" containerID="cri-o://40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50" gracePeriod=2 Nov 03 09:30:07.241906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.241869 4543 scope.go:122] "RemoveContainer" containerID="46678fd61a822f45d1f654b5fdfea37e06ad7bbe9e908b1621d44ad9bf7592e6" Nov 03 09:30:07.241953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.241916 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7387-21acf6" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" containerName="test-curl-7387-21acf6" containerID="cri-o://fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006" gracePeriod=2 Nov 03 09:30:07.345783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.345755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:07.390950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.390922 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-36a60c"] Nov 03 09:30:07.391495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391477 4543 cpu_manager.go:414] "containerMap: removing container" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" containerName="test-curl-7387-21acf6" Nov 03 09:30:07.391495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391494 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" containerName="test-curl-7387-21acf6" Nov 03 09:30:07.391644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391517 4543 cpu_manager.go:414] "containerMap: removing container" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" containerName="test-curl-7381-a57494" Nov 03 09:30:07.391644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391522 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" containerName="test-curl-7381-a57494" Nov 03 09:30:07.391644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391588 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" containerName="test-curl-7387-21acf6" Nov 03 09:30:07.391644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.391601 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" containerName="test-curl-7381-a57494" Nov 03 09:30:07.395948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.395929 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:07.450242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.450204 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-36a60c"] Nov 03 09:30:07.457769 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.457745 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlf6\" (UniqueName: \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\") pod \"test-curl-7381-36a60c\" (UID: \"4adc7e7b-90d0-4e64-96a7-994a0f0413fe\") " pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:07.523281 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.523242 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-21acf6\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:07.559121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.559042 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlf6\" (UniqueName: \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\") pod \"test-curl-7381-36a60c\" (UID: \"4adc7e7b-90d0-4e64-96a7-994a0f0413fe\") " pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:07.574856 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.574824 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a57494\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:07.618784 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.618762 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlf6\" (UniqueName: \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\") pod \"test-curl-7381-36a60c\" (UID: \"4adc7e7b-90d0-4e64-96a7-994a0f0413fe\") " pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:07.628486 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.626375 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-21acf6\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:07.677979 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.677947 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a57494\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:07.707897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.707873 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:07.765428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.765403 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:30:07.775377 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.773682 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.846343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.836507 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-03a6e5-openshift-default-dockercfg-49c6k\"" Nov 03 09:30:07.847711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.847679 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:30:07.850280 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.850244 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-21acf6\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864693 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9kr\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864760 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864789 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864809 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864825 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864848 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864866 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864906 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.866493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.864929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.902410 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:07.902368 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a57494\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:07.950835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.950784 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:07.965964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.965926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9kr\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966027 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966079 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966110 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966169 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966292 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966370 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966405 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966587 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.966675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.966636 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.967242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.967218 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.967429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.967407 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.968376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.968353 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.969469 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.969445 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:07.970168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:07.970122 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:08.029334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.029303 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:08.031060 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.031032 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9kr\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\") pod \"e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:08.090539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.090514 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-36a60c"] Nov 03 09:30:08.092101 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:08.092077 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4adc7e7b_90d0_4e64_96a7_994a0f0413fe.slice/crio-bd44ea98245b665a7a33f8ed493185327addf819a991032ce1b95ed840a29009 WatchSource:0}: Error finding container bd44ea98245b665a7a33f8ed493185327addf819a991032ce1b95ed840a29009: Status 404 returned error can't find the container with id bd44ea98245b665a7a33f8ed493185327addf819a991032ce1b95ed840a29009 Nov 03 09:30:08.103031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.102981 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:08.112814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.112796 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:08.256408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.256371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerStarted","Data":"6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67"} Nov 03 09:30:08.259177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.258669 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-36a60c" event={"ID":"4adc7e7b-90d0-4e64-96a7-994a0f0413fe","Type":"ContainerStarted","Data":"16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98"} Nov 03 09:30:08.259177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.258715 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-36a60c" event={"ID":"4adc7e7b-90d0-4e64-96a7-994a0f0413fe","Type":"ContainerStarted","Data":"bd44ea98245b665a7a33f8ed493185327addf819a991032ce1b95ed840a29009"} Nov 03 09:30:08.260803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.260780 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerStarted","Data":"f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9"} Nov 03 09:30:08.308236 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:08.308201 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-21acf6\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:08.478562 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:08.478526 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a57494\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:08.595062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.595005 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" podStartSLOduration=3.31332641 podStartE2EDuration="8.594992649s" totalImagesPullingTime="5.281666239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:08.593589822 +0000 UTC m=+3032.261630562" watchObservedRunningTime="2026-11-03 09:30:08.594992649 +0000 UTC m=+3032.263033406" Nov 03 09:30:08.604015 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.603962 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:30:08.606725 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:08.606688 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdaccee_0f81_4e30_b7c3_f3511386cd78.slice/crio-1803f052a946e3a63920ae544276f1f41e18de10d785ef8da3320250eb148518 WatchSource:0}: Error finding container 1803f052a946e3a63920ae544276f1f41e18de10d785ef8da3320250eb148518: Status 404 returned error can't find the container with id 1803f052a946e3a63920ae544276f1f41e18de10d785ef8da3320250eb148518 Nov 03 09:30:08.609658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.609625 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:08.609750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.609690 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:08.609750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.609719 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:08.711917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:08.711854 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-36a60c" podStartSLOduration=1.711834522 podStartE2EDuration="1.711834522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:08.708157931 +0000 UTC m=+3032.376198672" watchObservedRunningTime="2026-11-03 09:30:08.711834522 +0000 UTC m=+3032.379875262" Nov 03 09:30:09.269111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.269080 4543 generic.go:356] "Generic (PLEG): container finished" podID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerID="6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67" exitCode=60 Nov 03 09:30:09.269492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.269176 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerDied","Data":"6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67"} Nov 03 09:30:09.269492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.269243 4543 scope.go:122] "RemoveContainer" containerID="6c0d1022611892b0f9338bf4661290ed1d1a7f0041a0c042b229bc211a85c9e0" Nov 03 09:30:09.269605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.269535 4543 scope.go:122] "RemoveContainer" containerID="6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67" Nov 03 09:30:09.269921 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:09.269885 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk_odh-ai-gateway-infra(a2662c97-5fb7-4468-bc36-8a1b8367eb50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" Nov 03 09:30:09.271522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.271490 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerStarted","Data":"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5"} Nov 03 09:30:09.271656 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.271541 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerStarted","Data":"1803f052a946e3a63920ae544276f1f41e18de10d785ef8da3320250eb148518"} Nov 03 09:30:09.273755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.273733 4543 generic.go:356] "Generic (PLEG): container finished" podID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerID="f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9" exitCode=60 Nov 03 09:30:09.273850 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.273812 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerDied","Data":"f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9"} Nov 03 09:30:09.274009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.273993 4543 scope.go:122] "RemoveContainer" containerID="f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9" Nov 03 09:30:09.274257 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:09.274239 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch_odh-ai-gateway-infra(b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" Nov 03 09:30:09.349084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.349033 4543 scope.go:122] "RemoveContainer" containerID="46678fd61a822f45d1f654b5fdfea37e06ad7bbe9e908b1621d44ad9bf7592e6" Nov 03 09:30:09.507768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.507688 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podStartSLOduration=2.507673199 podStartE2EDuration="2.507673199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:09.504058552 +0000 UTC m=+3033.172099313" watchObservedRunningTime="2026-11-03 09:30:09.507673199 +0000 UTC m=+3033.175713937" Nov 03 09:30:09.507910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.507837 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:09.510785 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.510752 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:09.584899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.584872 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\" (UniqueName: \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\") pod \"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2\" (UID: \"c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2\") " Nov 03 09:30:09.585059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.584908 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\" (UniqueName: \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\") pod \"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6\" (UID: \"89ec5912-3ed0-40ef-a3c9-e9d30eccbae6\") " Nov 03 09:30:09.587197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.587166 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t" pod "c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" (UID: "c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2"). InnerVolumeSpecName "kube-api-access-7855t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:09.587310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.587236 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc" pod "89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" (UID: "89ec5912-3ed0-40ef-a3c9-e9d30eccbae6"). InnerVolumeSpecName "kube-api-access-jc7sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:09.685942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.685911 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7855t\" (UniqueName: \"kubernetes.io/projected/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2-kube-api-access-7855t\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:09.686109 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:09.685953 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc7sc\" (UniqueName: \"kubernetes.io/projected/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6-kube-api-access-jc7sc\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:10.100879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.100846 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-36a60c"] Nov 03 09:30:10.104788 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.104763 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-36a60c"] Nov 03 09:30:10.113300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.113274 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:10.119025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.119001 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:10.251195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.251113 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-392774"] Nov 03 09:30:10.251671 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.251656 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" containerName="test-curl-7381-36a60c" Nov 03 09:30:10.251727 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.251673 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" containerName="test-curl-7381-36a60c" Nov 03 09:30:10.251782 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.251773 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" containerName="test-curl-7381-36a60c" Nov 03 09:30:10.255216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.255202 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:10.280825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.280791 4543 generic.go:356] "Generic (PLEG): container finished" podID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" containerID="fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006" exitCode=137 Nov 03 09:30:10.281238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.280836 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:10.281238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.280899 4543 scope.go:122] "RemoveContainer" containerID="fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006" Nov 03 09:30:10.282305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.282281 4543 generic.go:356] "Generic (PLEG): container finished" podID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" containerID="40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50" exitCode=137 Nov 03 09:30:10.282398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.282307 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:10.289855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.289772 4543 scope.go:122] "RemoveContainer" containerID="6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67" Nov 03 09:30:10.290397 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.290073 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk_odh-ai-gateway-infra(a2662c97-5fb7-4468-bc36-8a1b8367eb50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" Nov 03 09:30:10.290516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.290451 4543 scope.go:122] "RemoveContainer" containerID="fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006" Nov 03 09:30:10.290810 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.290790 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006\": container with ID starting with fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006 not found: ID does not exist" containerID="fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006" Nov 03 09:30:10.290900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.290815 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006"} err="failed to get container status \"fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006\": rpc error: code = NotFound desc = could not find container \"fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006\": container with ID starting with fa7f5f8d8cbdcc075d8d2cfe0ae6bb8e97456a9d0794a819f3e778d70dce4006 not found: ID does not exist" Nov 03 09:30:10.290900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.290833 4543 scope.go:122] "RemoveContainer" containerID="40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50" Nov 03 09:30:10.291550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.291524 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-36a60c" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" containerName="test-curl-7381-36a60c" containerID="cri-o://16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98" gracePeriod=2 Nov 03 09:30:10.291550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.291546 4543 scope.go:122] "RemoveContainer" containerID="f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9" Nov 03 09:30:10.291838 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.291763 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:10.291838 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.291796 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch_odh-ai-gateway-infra(b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" Nov 03 09:30:10.292025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.291960 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp8pr\" (UniqueName: \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\") pod \"test-curl-7381-392774\" (UID: \"94e36853-bb40-4120-aaa6-0ee50306d18e\") " pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:10.293266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.293244 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:30:10.300414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.300394 4543 scope.go:122] "RemoveContainer" containerID="40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50" Nov 03 09:30:10.300679 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.300660 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50\": container with ID starting with 40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50 not found: ID does not exist" containerID="40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50" Nov 03 09:30:10.300742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.300685 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50"} err="failed to get container status \"40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50\": rpc error: code = NotFound desc = could not find container \"40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50\": container with ID starting with 40cc4b3da4626aa1b913ec377e7f75ce0f7777f43ee2d2bdfe7b7c633fe68a50 not found: ID does not exist" Nov 03 09:30:10.314417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.314390 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-392774"] Nov 03 09:30:10.393429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.393395 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp8pr\" (UniqueName: \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\") pod \"test-curl-7381-392774\" (UID: \"94e36853-bb40-4120-aaa6-0ee50306d18e\") " pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:10.421153 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.421108 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a57494\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" pod="opendatahub/test-curl-7381-a57494" Nov 03 09:30:10.451575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.451537 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ec5912-3ed0-40ef-a3c9-e9d30eccbae6" path="/var/lib/kubelet/pods/89ec5912-3ed0-40ef-a3c9-e9d30eccbae6/volumes" Nov 03 09:30:10.451932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.451916 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" path="/var/lib/kubelet/pods/c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2/volumes" Nov 03 09:30:10.453134 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.453112 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp8pr\" (UniqueName: \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\") pod \"test-curl-7381-392774\" (UID: \"94e36853-bb40-4120-aaa6-0ee50306d18e\") " pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:10.472790 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.472761 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-36a60c\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:10.567412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.567341 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:10.758785 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:10.758745 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-21acf6\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="c5da36a7-fa55-4cd1-829c-a4e1f9c70ed2" pod="opendatahub/test-curl-7387-21acf6" Nov 03 09:30:10.961181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:10.955705 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-392774"] Nov 03 09:30:10.965009 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:10.964979 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e36853_bb40_4120_aaa6_0ee50306d18e.slice/crio-db473a1d9eecf48e0c516c69c356184853d8b83ec693cf9ea83b15cc2b655e87 WatchSource:0}: Error finding container db473a1d9eecf48e0c516c69c356184853d8b83ec693cf9ea83b15cc2b655e87: Status 404 returned error can't find the container with id db473a1d9eecf48e0c516c69c356184853d8b83ec693cf9ea83b15cc2b655e87 Nov 03 09:30:11.300041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:11.299943 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-392774" event={"ID":"94e36853-bb40-4120-aaa6-0ee50306d18e","Type":"ContainerStarted","Data":"5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee"} Nov 03 09:30:11.300041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:11.299991 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-392774" event={"ID":"94e36853-bb40-4120-aaa6-0ee50306d18e","Type":"ContainerStarted","Data":"db473a1d9eecf48e0c516c69c356184853d8b83ec693cf9ea83b15cc2b655e87"} Nov 03 09:30:11.416087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:11.416025 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-392774" podStartSLOduration=1.416011607 podStartE2EDuration="1.416011607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:11.414024283 +0000 UTC m=+3035.082065023" watchObservedRunningTime="2026-11-03 09:30:11.416011607 +0000 UTC m=+3035.084052345" Nov 03 09:30:12.358854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.358821 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:12.509083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.509014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:12.514815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.514795 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:12.621346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.621316 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\" (UniqueName: \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\") pod \"4adc7e7b-90d0-4e64-96a7-994a0f0413fe\" (UID: \"4adc7e7b-90d0-4e64-96a7-994a0f0413fe\") " Nov 03 09:30:12.623578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.623542 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6" pod "4adc7e7b-90d0-4e64-96a7-994a0f0413fe" (UID: "4adc7e7b-90d0-4e64-96a7-994a0f0413fe"). InnerVolumeSpecName "kube-api-access-tqlf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:12.646564 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.646536 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:12.723273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.723245 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlf6\" (UniqueName: \"kubernetes.io/projected/4adc7e7b-90d0-4e64-96a7-994a0f0413fe-kube-api-access-tqlf6\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:12.881523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.881429 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-392774"] Nov 03 09:30:12.884679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:12.884651 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-392774"] Nov 03 09:30:13.025925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.025885 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-42c503"] Nov 03 09:30:13.026498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.026480 4543 cpu_manager.go:414] "containerMap: removing container" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" containerName="test-curl-7381-392774" Nov 03 09:30:13.026559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.026501 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" containerName="test-curl-7381-392774" Nov 03 09:30:13.026631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.026620 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" containerName="test-curl-7381-392774" Nov 03 09:30:13.030189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.030173 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:13.084133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.084102 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-42c503"] Nov 03 09:30:13.127959 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.127927 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zmz\" (UniqueName: \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\") pod \"test-curl-7381-42c503\" (UID: \"6b20ba5d-0619-466e-876a-5ff3bfd66775\") " pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:13.228793 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.228709 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zmz\" (UniqueName: \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\") pod \"test-curl-7381-42c503\" (UID: \"6b20ba5d-0619-466e-876a-5ff3bfd66775\") " pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:13.291477 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.291443 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zmz\" (UniqueName: \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\") pod \"test-curl-7381-42c503\" (UID: \"6b20ba5d-0619-466e-876a-5ff3bfd66775\") " pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:13.314983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.314955 4543 generic.go:356] "Generic (PLEG): container finished" podID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" containerID="16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98" exitCode=137 Nov 03 09:30:13.315172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.314996 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:13.315172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.315060 4543 scope.go:122] "RemoveContainer" containerID="16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98" Nov 03 09:30:13.315294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.315271 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-392774" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" containerName="test-curl-7381-392774" containerID="cri-o://5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee" gracePeriod=2 Nov 03 09:30:13.324330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.324314 4543 scope.go:122] "RemoveContainer" containerID="16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98" Nov 03 09:30:13.324595 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:13.324580 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98\": container with ID starting with 16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98 not found: ID does not exist" containerID="16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98" Nov 03 09:30:13.324647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.324604 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98"} err="failed to get container status \"16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98\": rpc error: code = NotFound desc = could not find container \"16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98\": container with ID starting with 16a3d7654679ca3c490a02e84edaa373be4835a90903ab18743ae38ab5953a98 not found: ID does not exist" Nov 03 09:30:13.343588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.343567 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:13.366899 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:13.366859 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-392774\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:13.418444 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:13.418398 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-36a60c\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:13.469825 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:13.469790 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-392774\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:13.720115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:13.720086 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-42c503"] Nov 03 09:30:13.721492 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:13.721468 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b20ba5d_0619_466e_876a_5ff3bfd66775.slice/crio-b349926feb8844d632c87bd1d6b4c0e65a8b0d78e23af0d863c4975c2ab5daf4 WatchSource:0}: Error finding container b349926feb8844d632c87bd1d6b4c0e65a8b0d78e23af0d863c4975c2ab5daf4: Status 404 returned error can't find the container with id b349926feb8844d632c87bd1d6b4c0e65a8b0d78e23af0d863c4975c2ab5daf4 Nov 03 09:30:14.324833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:14.324782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-42c503" event={"ID":"6b20ba5d-0619-466e-876a-5ff3bfd66775","Type":"ContainerStarted","Data":"fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb"} Nov 03 09:30:14.324833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:14.324831 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-42c503" event={"ID":"6b20ba5d-0619-466e-876a-5ff3bfd66775","Type":"ContainerStarted","Data":"b349926feb8844d632c87bd1d6b4c0e65a8b0d78e23af0d863c4975c2ab5daf4"} Nov 03 09:30:14.377115 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:14.377070 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-36a60c\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" pod="opendatahub/test-curl-7381-36a60c" Nov 03 09:30:14.451701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:14.451664 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adc7e7b-90d0-4e64-96a7-994a0f0413fe" path="/var/lib/kubelet/pods/4adc7e7b-90d0-4e64-96a7-994a0f0413fe/volumes" Nov 03 09:30:14.492481 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:14.492434 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-42c503" podStartSLOduration=2.492419446 podStartE2EDuration="2.492419446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:14.491411118 +0000 UTC m=+3038.159451857" watchObservedRunningTime="2026-11-03 09:30:14.492419446 +0000 UTC m=+3038.160460184" Nov 03 09:30:14.542748 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:14.542714 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-392774\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:15.565895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.565868 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:15.580816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.580756 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:15.658627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.658590 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\" (UniqueName: \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\") pod \"94e36853-bb40-4120-aaa6-0ee50306d18e\" (UID: \"94e36853-bb40-4120-aaa6-0ee50306d18e\") " Nov 03 09:30:15.660739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.660718 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr" pod "94e36853-bb40-4120-aaa6-0ee50306d18e" (UID: "94e36853-bb40-4120-aaa6-0ee50306d18e"). InnerVolumeSpecName "kube-api-access-mp8pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:15.760067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.760019 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp8pr\" (UniqueName: \"kubernetes.io/projected/94e36853-bb40-4120-aaa6-0ee50306d18e-kube-api-access-mp8pr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:15.812476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.812429 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:15.839725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.839640 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-42c503"] Nov 03 09:30:15.842842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.842809 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-42c503"] Nov 03 09:30:15.984640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.984596 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-a50475"] Nov 03 09:30:15.985269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.985246 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" containerName="test-curl-7381-42c503" Nov 03 09:30:15.985269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.985268 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" containerName="test-curl-7381-42c503" Nov 03 09:30:15.985446 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.985426 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" containerName="test-curl-7381-42c503" Nov 03 09:30:15.988691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:15.988670 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:16.044353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.044319 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-a50475"] Nov 03 09:30:16.063244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.063211 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24d4l\" (UniqueName: \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\") pod \"test-curl-7381-a50475\" (UID: \"bdeab4fe-c6dc-4bff-b951-f075215f119d\") " pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:16.185605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.179956 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24d4l\" (UniqueName: \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\") pod \"test-curl-7381-a50475\" (UID: \"bdeab4fe-c6dc-4bff-b951-f075215f119d\") " pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:16.246128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.246095 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24d4l\" (UniqueName: \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\") pod \"test-curl-7381-a50475\" (UID: \"bdeab4fe-c6dc-4bff-b951-f075215f119d\") " pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:16.300882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.300846 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:16.339467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.339428 4543 generic.go:356] "Generic (PLEG): container finished" podID="94e36853-bb40-4120-aaa6-0ee50306d18e" containerID="5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee" exitCode=137 Nov 03 09:30:16.339624 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.339477 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:16.339624 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.339527 4543 scope.go:122] "RemoveContainer" containerID="5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee" Nov 03 09:30:16.339815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.339792 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-42c503" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" containerName="test-curl-7381-42c503" containerID="cri-o://fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb" gracePeriod=2 Nov 03 09:30:16.358955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.358712 4543 scope.go:122] "RemoveContainer" containerID="5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee" Nov 03 09:30:16.361111 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:16.360897 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee\": container with ID starting with 5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee not found: ID does not exist" containerID="5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee" Nov 03 09:30:16.361111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.360941 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee"} err="failed to get container status \"5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee\": rpc error: code = NotFound desc = could not find container \"5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee\": container with ID starting with 5090dec0d9556ffb737ed10850406ce814e57bbaef4374d5dc7cdffe54f70eee not found: ID does not exist" Nov 03 09:30:16.391565 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:16.391524 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-42c503\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:16.442988 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:16.442902 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-42c503\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:16.451621 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.451593 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" path="/var/lib/kubelet/pods/94e36853-bb40-4120-aaa6-0ee50306d18e/volumes" Nov 03 09:30:16.494359 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:16.494321 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-392774\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="94e36853-bb40-4120-aaa6-0ee50306d18e" pod="opendatahub/test-curl-7381-392774" Nov 03 09:30:16.684416 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:16.684392 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-a50475"] Nov 03 09:30:16.685581 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:16.685552 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdeab4fe_c6dc_4bff_b951_f075215f119d.slice/crio-9856ea370461dac55d4deacdd3d1add7df919dd898d5fb23600e6416f0213dd0 WatchSource:0}: Error finding container 9856ea370461dac55d4deacdd3d1add7df919dd898d5fb23600e6416f0213dd0: Status 404 returned error can't find the container with id 9856ea370461dac55d4deacdd3d1add7df919dd898d5fb23600e6416f0213dd0 Nov 03 09:30:17.348583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:17.348545 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-a50475" event={"ID":"bdeab4fe-c6dc-4bff-b951-f075215f119d","Type":"ContainerStarted","Data":"53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3"} Nov 03 09:30:17.348583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:17.348588 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-a50475" event={"ID":"bdeab4fe-c6dc-4bff-b951-f075215f119d","Type":"ContainerStarted","Data":"9856ea370461dac55d4deacdd3d1add7df919dd898d5fb23600e6416f0213dd0"} Nov 03 09:30:17.467709 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:17.467654 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-a50475" podStartSLOduration=2.467638406 podStartE2EDuration="2.467638406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:17.466190628 +0000 UTC m=+3041.134231368" watchObservedRunningTime="2026-11-03 09:30:17.467638406 +0000 UTC m=+3041.135679145" Nov 03 09:30:17.602641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:17.602560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:18.578618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.578585 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:18.631602 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:18.631564 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-42c503\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:18.725275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.725248 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\" (UniqueName: \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\") pod \"6b20ba5d-0619-466e-876a-5ff3bfd66775\" (UID: \"6b20ba5d-0619-466e-876a-5ff3bfd66775\") " Nov 03 09:30:18.727428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.727395 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz" pod "6b20ba5d-0619-466e-876a-5ff3bfd66775" (UID: "6b20ba5d-0619-466e-876a-5ff3bfd66775"). InnerVolumeSpecName "kube-api-access-q8zmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:18.826499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.826466 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zmz\" (UniqueName: \"kubernetes.io/projected/6b20ba5d-0619-466e-876a-5ff3bfd66775-kube-api-access-q8zmz\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:18.924789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.924706 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-a50475"] Nov 03 09:30:18.932215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:18.932184 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-a50475"] Nov 03 09:30:19.363656 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.363622 4543 generic.go:356] "Generic (PLEG): container finished" podID="6b20ba5d-0619-466e-876a-5ff3bfd66775" containerID="fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb" exitCode=137 Nov 03 09:30:19.363835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.363666 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:19.363835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.363732 4543 scope.go:122] "RemoveContainer" containerID="fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb" Nov 03 09:30:19.363936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.363861 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-a50475" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" containerName="test-curl-7381-a50475" containerID="cri-o://53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3" gracePeriod=2 Nov 03 09:30:19.374499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.374478 4543 scope.go:122] "RemoveContainer" containerID="fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb" Nov 03 09:30:19.374760 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:19.374744 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb\": container with ID starting with fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb not found: ID does not exist" containerID="fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb" Nov 03 09:30:19.374818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:19.374769 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb"} err="failed to get container status \"fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb\": rpc error: code = NotFound desc = could not find container \"fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb\": container with ID starting with fdd7c56a724e888b5488008c24bfeec859d01d53333a59bf12e72a549332f4bb not found: ID does not exist" Nov 03 09:30:19.415814 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:19.415775 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a50475\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:19.467765 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:19.467737 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-42c503\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:19.519458 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:19.519427 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-a50475\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:19.571183 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:19.571121 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-42c503\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" pod="opendatahub/test-curl-7381-42c503" Nov 03 09:30:20.417930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.417890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:20.451944 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.451915 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b20ba5d-0619-466e-876a-5ff3bfd66775" path="/var/lib/kubelet/pods/6b20ba5d-0619-466e-876a-5ff3bfd66775/volumes" Nov 03 09:30:20.916331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.916275 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:30:20.916868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.916853 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" containerName="test-curl-7381-a50475" Nov 03 09:30:20.916940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.916870 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" containerName="test-curl-7381-a50475" Nov 03 09:30:20.916992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.916983 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" containerName="test-curl-7381-a50475" Nov 03 09:30:20.921627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:20.921605 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.033172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.031393 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:30:21.038826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.038791 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:21.038992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.038977 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-739a7f-openshift-default-dockercfg-r98km\"" Nov 03 09:30:21.052000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.051964 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052014 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7rm\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052106 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052343 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052399 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052451 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052551 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052695 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.052805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.052753 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154255 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154327 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154355 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154409 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154429 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7rm\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154447 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154505 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154545 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154592 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154645 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154876 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.154963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.154920 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.155206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.155185 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.155316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.155299 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.157526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.157503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.157843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.157823 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.211314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.211226 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:21.214653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.214619 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.214653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.214647 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7rm\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\") pod \"e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.235599 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.235566 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:21.444725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.444697 4543 scope.go:122] "RemoveContainer" containerID="f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9" Nov 03 09:30:21.445124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.444738 4543 scope.go:122] "RemoveContainer" containerID="6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67" Nov 03 09:30:21.570381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.570353 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:21.677129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.677103 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:30:21.683526 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:21.683499 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8f298c_75bd_4615_9aa5_49fe66f7b97f.slice/crio-e7103c0342f4bf6391e1cebb51f9e81152762cfecffd7a43b38e9b97fc995450 WatchSource:0}: Error finding container e7103c0342f4bf6391e1cebb51f9e81152762cfecffd7a43b38e9b97fc995450: Status 404 returned error can't find the container with id e7103c0342f4bf6391e1cebb51f9e81152762cfecffd7a43b38e9b97fc995450 Nov 03 09:30:21.686923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.686892 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:21.687020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.686965 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:21.687020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.686996 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:21.763526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.763488 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\" (UniqueName: \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\") pod \"bdeab4fe-c6dc-4bff-b951-f075215f119d\" (UID: \"bdeab4fe-c6dc-4bff-b951-f075215f119d\") " Nov 03 09:30:21.765801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.765767 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l" pod "bdeab4fe-c6dc-4bff-b951-f075215f119d" (UID: "bdeab4fe-c6dc-4bff-b951-f075215f119d"). InnerVolumeSpecName "kube-api-access-24d4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:21.865845 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:21.865819 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24d4l\" (UniqueName: \"kubernetes.io/projected/bdeab4fe-c6dc-4bff-b951-f075215f119d-kube-api-access-24d4l\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:22.385247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.385208 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerStarted","Data":"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6"} Nov 03 09:30:22.385415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.385306 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerStarted","Data":"e7103c0342f4bf6391e1cebb51f9e81152762cfecffd7a43b38e9b97fc995450"} Nov 03 09:30:22.387121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.386924 4543 generic.go:356] "Generic (PLEG): container finished" podID="bdeab4fe-c6dc-4bff-b951-f075215f119d" containerID="53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3" exitCode=137 Nov 03 09:30:22.387121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.387049 4543 scope.go:122] "RemoveContainer" containerID="53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3" Nov 03 09:30:22.387121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.387011 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-a50475" Nov 03 09:30:22.390423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.390393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerStarted","Data":"85e5a9bbedb55e73a7ce51c5b4a764ee3e1a7d32e61fff6c71a719a401e929fa"} Nov 03 09:30:22.392927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.392905 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerStarted","Data":"cb45dadc19f49676f18c26bd43f854973a9f7231e996dbd40e25e46b2e306acf"} Nov 03 09:30:22.452566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.452490 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdeab4fe-c6dc-4bff-b951-f075215f119d" path="/var/lib/kubelet/pods/bdeab4fe-c6dc-4bff-b951-f075215f119d/volumes" Nov 03 09:30:22.459102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.459079 4543 scope.go:122] "RemoveContainer" containerID="53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3" Nov 03 09:30:22.459423 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:22.459404 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3\": container with ID starting with 53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3 not found: ID does not exist" containerID="53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3" Nov 03 09:30:22.459501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.459442 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3"} err="failed to get container status \"53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3\": rpc error: code = NotFound desc = could not find container \"53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3\": container with ID starting with 53c772bc986310cccddd6fc442bd5a291cd4e06fadd6f696fab6e17b43f92bc3 not found: ID does not exist" Nov 03 09:30:22.508905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.508856 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podStartSLOduration=2.508841787 podStartE2EDuration="2.508841787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:22.506821545 +0000 UTC m=+3046.174862284" watchObservedRunningTime="2026-11-03 09:30:22.508841787 +0000 UTC m=+3046.176882525" Nov 03 09:30:22.760921 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:22.760814 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:23.236370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.236336 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:23.242339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.242316 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:23.401129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.401092 4543 generic.go:356] "Generic (PLEG): container finished" podID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerID="85e5a9bbedb55e73a7ce51c5b4a764ee3e1a7d32e61fff6c71a719a401e929fa" exitCode=60 Nov 03 09:30:23.401325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.401165 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerDied","Data":"85e5a9bbedb55e73a7ce51c5b4a764ee3e1a7d32e61fff6c71a719a401e929fa"} Nov 03 09:30:23.401325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.401225 4543 scope.go:122] "RemoveContainer" containerID="6ec27d916ab5339dd30fb65d4e8a5af8c6463531ea3bfb41190c40bad5021c67" Nov 03 09:30:23.401593 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.401574 4543 scope.go:122] "RemoveContainer" containerID="85e5a9bbedb55e73a7ce51c5b4a764ee3e1a7d32e61fff6c71a719a401e929fa" Nov 03 09:30:23.401915 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:23.401886 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk_odh-ai-gateway-infra(a2662c97-5fb7-4468-bc36-8a1b8367eb50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" Nov 03 09:30:23.403321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.403301 4543 generic.go:356] "Generic (PLEG): container finished" podID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerID="cb45dadc19f49676f18c26bd43f854973a9f7231e996dbd40e25e46b2e306acf" exitCode=60 Nov 03 09:30:23.403414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.403370 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerDied","Data":"cb45dadc19f49676f18c26bd43f854973a9f7231e996dbd40e25e46b2e306acf"} Nov 03 09:30:23.403688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.403674 4543 scope.go:122] "RemoveContainer" containerID="cb45dadc19f49676f18c26bd43f854973a9f7231e996dbd40e25e46b2e306acf" Nov 03 09:30:23.403935 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:23.403919 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch_odh-ai-gateway-infra(b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" Nov 03 09:30:23.404687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.404665 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:23.405750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.405732 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:30:23.414426 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.414407 4543 scope.go:122] "RemoveContainer" containerID="f8710421f7cbe86de5ac75bfa1e6f8b1e2b118b5f6af31b59544150da3a445b9" Nov 03 09:30:23.636319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.636286 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:23.757763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:23.757723 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:24.569982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.569956 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:24.594430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.594409 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:24.596798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.596775 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\" (UniqueName: \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\") pod \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " Nov 03 09:30:24.596899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.596840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\") pod \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\" (UID: \"a2662c97-5fb7-4468-bc36-8a1b8367eb50\") " Nov 03 09:30:24.597211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.597191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca" pod "a2662c97-5fb7-4468-bc36-8a1b8367eb50" (UID: "a2662c97-5fb7-4468-bc36-8a1b8367eb50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:30:24.597334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.597320 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2662c97-5fb7-4468-bc36-8a1b8367eb50-maas-api-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:24.599113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.599090 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9" pod "a2662c97-5fb7-4468-bc36-8a1b8367eb50" (UID: "a2662c97-5fb7-4468-bc36-8a1b8367eb50"). InnerVolumeSpecName "kube-api-access-7x4h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:24.698571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.698453 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\" (UniqueName: \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\") pod \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " Nov 03 09:30:24.698947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.698599 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\") pod \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\" (UID: \"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca\") " Nov 03 09:30:24.698947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.698911 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca" pod "b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" (UID: "b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:30:24.699051 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.699038 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4h9\" (UniqueName: \"kubernetes.io/projected/a2662c97-5fb7-4468-bc36-8a1b8367eb50-kube-api-access-7x4h9\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:24.699090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.699054 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-maas-api-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:24.700802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.700783 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm" pod "b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" (UID: "b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca"). InnerVolumeSpecName "kube-api-access-ptqlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:30:24.800596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:24.800557 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqlm\" (UniqueName: \"kubernetes.io/projected/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca-kube-api-access-ptqlm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:30:25.254546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.254507 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:25.422026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.421994 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" Nov 03 09:30:25.422026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.422013 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk" event={"ID":"a2662c97-5fb7-4468-bc36-8a1b8367eb50","Type":"ContainerDied","Data":"e9bc76bf3d3877a3bc0187f28d44a6e3773ef25ce13dd661bbfa96bccb8ed6c3"} Nov 03 09:30:25.422307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.422089 4543 scope.go:122] "RemoveContainer" containerID="85e5a9bbedb55e73a7ce51c5b4a764ee3e1a7d32e61fff6c71a719a401e929fa" Nov 03 09:30:25.423674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.423656 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" Nov 03 09:30:25.423777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.423676 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch" event={"ID":"b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca","Type":"ContainerDied","Data":"3a8a83a5d175f876ad38c3975f36cf448c143f95a48c4a315dc36fd459b650ab"} Nov 03 09:30:25.432339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.432312 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:25.433047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.432896 4543 scope.go:122] "RemoveContainer" containerID="cb45dadc19f49676f18c26bd43f854973a9f7231e996dbd40e25e46b2e306acf" Nov 03 09:30:25.572271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.572235 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:25.591377 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.591317 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:25.641677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.641637 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-506e6d-29894970-7hlch"] Nov 03 09:30:25.709504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.709473 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:25.719609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:25.719582 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894970-l2twk"] Nov 03 09:30:26.451818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:26.451787 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" path="/var/lib/kubelet/pods/a2662c97-5fb7-4468-bc36-8a1b8367eb50/volumes" Nov 03 09:30:26.452220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:26.452206 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" path="/var/lib/kubelet/pods/b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca/volumes" Nov 03 09:30:28.182494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:28.182460 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:28.551746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:28.551662 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:28.736050 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:28.736015 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:31.724770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:31.724737 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:32.249060 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249021 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:30:32.249718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249696 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249718 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249742 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249750 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249760 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249766 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249788 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249796 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249840 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.249923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249847 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249971 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249988 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.249995 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b786bf7d-7b2e-41d5-a6ae-4fcc66a18dca" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.250009 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.250021 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.250362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.250032 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:32.254947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.254926 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.308895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.308849 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.310172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.310124 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.310467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.310446 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.310674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.310654 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67z5s\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.310942 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.310925 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.316375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.316353 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.317554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.317534 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.317757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.317730 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.317966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.317950 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.318408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.318390 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f66c6c-openshift-default-dockercfg-vx98r\"" Nov 03 09:30:32.340336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.340288 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:32.366893 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.366851 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.425974 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.426125 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.427947 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.427977 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.428048 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.428080 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.428104 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.428798 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.427660 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.429031 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.427878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.429456 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.429515 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67z5s\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.432221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.430462 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.441777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.441755 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.442501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.442479 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.489091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.489061 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.490397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.490373 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67z5s\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\") pod \"e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.569496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.569402 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:32.654280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:32.654041 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:33.001664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.001632 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:30:33.002737 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:33.002710 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4399c8_4654_49b1_82cd_a6d22f04d919.slice/crio-950959ba6d5df62d6e01090a19bf7eef3671da3ba4e2c8dd8374cd6e90bba74b WatchSource:0}: Error finding container 950959ba6d5df62d6e01090a19bf7eef3671da3ba4e2c8dd8374cd6e90bba74b: Status 404 returned error can't find the container with id 950959ba6d5df62d6e01090a19bf7eef3671da3ba4e2c8dd8374cd6e90bba74b Nov 03 09:30:33.005663 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.005629 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:33.005755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.005701 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:33.005755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.005731 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:33.088221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.088194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:33.332689 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.332606 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:33.484432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.484398 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerStarted","Data":"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5"} Nov 03 09:30:33.484611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.484451 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerStarted","Data":"950959ba6d5df62d6e01090a19bf7eef3671da3ba4e2c8dd8374cd6e90bba74b"} Nov 03 09:30:33.570344 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.570310 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:33.621084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:33.617867 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podStartSLOduration=1.617847255 podStartE2EDuration="1.617847255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:33.616620206 +0000 UTC m=+3057.284660945" watchObservedRunningTime="2026-11-03 09:30:33.617847255 +0000 UTC m=+3057.285887996" Nov 03 09:30:34.575864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:34.575836 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:35.495995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.495963 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:35.496852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.496834 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:30:35.561291 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:30:35.561253 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:30:35.808807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.808732 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:30:35.809406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.809389 4543 cpu_manager.go:414] "containerMap: removing container" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:35.809470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.809408 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2662c97-5fb7-4468-bc36-8a1b8367eb50" containerName="cleanup" Nov 03 09:30:35.812895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.812875 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.867762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.867733 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f01a6d-openshift-default-dockercfg-l52zf\"" Nov 03 09:30:35.872634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.872594 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.872923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.872899 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873052 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.872940 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873052 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.872996 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.873626 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.873709 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.873767 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.873797 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59c9\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.873953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.873846 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.874741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.874721 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:30:35.900075 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.900046 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:35.986572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.986355 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.986572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.986493 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.986997 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987168 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987055 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59c9\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987400 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987498 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987588 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987632 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987745 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.987946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.987894 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.988900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.988875 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.994652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.994622 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.994822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.994805 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.994976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.994962 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.995399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.995382 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:35.998267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:35.998245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:36.047631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.047596 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:36.053659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.053630 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59c9\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:36.128229 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.128197 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:36.217167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.215589 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:36.360591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.360553 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:36.563163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.563067 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:36.564637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.564608 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:30:36.599007 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:30:36.598981 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49d17b5_47ee_4910_81f0_8b8ac0d56cad.slice/crio-136de4bc053c5536d691f6428fd9aaec103ce848315c5297dee85c7d055cd3e0 WatchSource:0}: Error finding container 136de4bc053c5536d691f6428fd9aaec103ce848315c5297dee85c7d055cd3e0: Status 404 returned error can't find the container with id 136de4bc053c5536d691f6428fd9aaec103ce848315c5297dee85c7d055cd3e0 Nov 03 09:30:36.601581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.601552 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:36.601640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.601618 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:36.601684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.601645 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:30:36.701841 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:36.701810 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:37.510129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:37.510040 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerStarted","Data":"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06"} Nov 03 09:30:37.510129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:37.510099 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerStarted","Data":"136de4bc053c5536d691f6428fd9aaec103ce848315c5297dee85c7d055cd3e0"} Nov 03 09:30:37.688001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:37.687949 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podStartSLOduration=2.687934902 podStartE2EDuration="2.687934902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:37.685294271 +0000 UTC m=+3061.353335009" watchObservedRunningTime="2026-11-03 09:30:37.687934902 +0000 UTC m=+3061.355975640" Nov 03 09:30:38.128960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:38.128927 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:38.134964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:38.134935 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:38.516273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:38.516189 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:38.517331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:38.517313 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:30:38.746090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:38.746050 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:39.627501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:39.627466 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:39.816746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:39.816716 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:41.180198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:41.180162 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:41.355725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:41.355651 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:41.562566 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:41.562485 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:41.703326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:41.703296 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:42.755334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:42.755299 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:44.142682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:44.142647 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:44.350951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:44.350922 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:45.113463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:45.113421 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:45.243693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:45.243657 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:47.072222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:47.072185 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:47.258789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:47.258755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:49.448539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:49.448501 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:52.443363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:52.443321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:54.441332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:54.441293 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:56.288286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:56.288253 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:56.425330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:56.425292 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:30:59.453219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:30:59.453181 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:01.467524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.467489 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7375-700f6a"] Nov 03 09:31:01.474995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.474971 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:01.525209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.525175 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-700f6a"] Nov 03 09:31:01.549630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.549547 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vv5\" (UniqueName: \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\") pod \"test-curl-7375-700f6a\" (UID: \"e5b369c9-b7c9-48e6-84bc-926b2182250c\") " pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:01.572274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.572249 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-77bkh\"" Nov 03 09:31:01.650934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.650901 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vv5\" (UniqueName: \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\") pod \"test-curl-7375-700f6a\" (UID: \"e5b369c9-b7c9-48e6-84bc-926b2182250c\") " pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:01.716664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.716632 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vv5\" (UniqueName: \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\") pod \"test-curl-7375-700f6a\" (UID: \"e5b369c9-b7c9-48e6-84bc-926b2182250c\") " pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:01.788295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:01.788210 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:02.161619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:02.161594 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-700f6a"] Nov 03 09:31:02.163185 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:31:02.163130 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5b369c9_b7c9_48e6_84bc_926b2182250c.slice/crio-b6361478210047ea5f1648ee8dc80926a1f26b60b02a153da33a822b0e2b2c7e WatchSource:0}: Error finding container b6361478210047ea5f1648ee8dc80926a1f26b60b02a153da33a822b0e2b2c7e: Status 404 returned error can't find the container with id b6361478210047ea5f1648ee8dc80926a1f26b60b02a153da33a822b0e2b2c7e Nov 03 09:31:02.671086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:02.671034 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-700f6a" event={"ID":"e5b369c9-b7c9-48e6-84bc-926b2182250c","Type":"ContainerStarted","Data":"cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712"} Nov 03 09:31:02.671086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:02.671089 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-700f6a" event={"ID":"e5b369c9-b7c9-48e6-84bc-926b2182250c","Type":"ContainerStarted","Data":"b6361478210047ea5f1648ee8dc80926a1f26b60b02a153da33a822b0e2b2c7e"} Nov 03 09:31:02.786802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:02.786752 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7375-700f6a" podStartSLOduration=1.78673948 podStartE2EDuration="1.78673948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:31:02.784452391 +0000 UTC m=+3086.452493129" watchObservedRunningTime="2026-11-03 09:31:02.78673948 +0000 UTC m=+3086.454780219" Nov 03 09:31:04.170661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:04.170618 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7375-700f6a"] Nov 03 09:31:04.174294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:04.174266 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7375-700f6a"] Nov 03 09:31:04.409880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:04.409841 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:04.684428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:04.684387 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7375-700f6a" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" containerName="test-curl-7375-700f6a" containerID="cri-o://cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712" gracePeriod=2 Nov 03 09:31:05.562256 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:05.562218 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:31:06.935009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:06.934982 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:07.014820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.014741 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\" (UniqueName: \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\") pod \"e5b369c9-b7c9-48e6-84bc-926b2182250c\" (UID: \"e5b369c9-b7c9-48e6-84bc-926b2182250c\") " Nov 03 09:31:07.016962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.016932 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5" pod "e5b369c9-b7c9-48e6-84bc-926b2182250c" (UID: "e5b369c9-b7c9-48e6-84bc-926b2182250c"). InnerVolumeSpecName "kube-api-access-m4vv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:31:07.116310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.116267 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vv5\" (UniqueName: \"kubernetes.io/projected/e5b369c9-b7c9-48e6-84bc-926b2182250c-kube-api-access-m4vv5\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:31:07.703882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.703851 4543 generic.go:356] "Generic (PLEG): container finished" podID="e5b369c9-b7c9-48e6-84bc-926b2182250c" containerID="cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712" exitCode=137 Nov 03 09:31:07.704085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.703903 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:07.704085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.703953 4543 scope.go:122] "RemoveContainer" containerID="cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712" Nov 03 09:31:07.713267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.713248 4543 scope.go:122] "RemoveContainer" containerID="cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712" Nov 03 09:31:07.713514 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:07.713494 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712\": container with ID starting with cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712 not found: ID does not exist" containerID="cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712" Nov 03 09:31:07.713576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:07.713528 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712"} err="failed to get container status \"cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712\": rpc error: code = NotFound desc = could not find container \"cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712\": container with ID starting with cdf2f43ff44c69e4e991f73aa2cbd81c04cf4febe53d6bf49ffdbfe118537712 not found: ID does not exist" Nov 03 09:31:07.763746 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:07.763711 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7375-700f6a\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" pod="opendatahub/test-curl-7375-700f6a" Nov 03 09:31:08.452607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:08.452567 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" path="/var/lib/kubelet/pods/e5b369c9-b7c9-48e6-84bc-926b2182250c/volumes" Nov 03 09:31:09.433675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:09.433632 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:14.399565 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:14.399509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:19.374262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:19.374168 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:24.032848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.032816 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:31:24.033805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.033782 4543 cpu_manager.go:414] "containerMap: removing container" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" containerName="test-curl-7375-700f6a" Nov 03 09:31:24.033891 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.033807 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" containerName="test-curl-7375-700f6a" Nov 03 09:31:24.033931 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.033918 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b369c9-b7c9-48e6-84bc-926b2182250c" containerName="test-curl-7375-700f6a" Nov 03 09:31:24.038637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.038618 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.094626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.094592 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:31:24.112649 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.112618 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.112816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.112672 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.112888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.112831 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.112945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.112914 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk58b\" (UniqueName: \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.141039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.141010 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-03a6e5-serving-cert\"" Nov 03 09:31:24.214510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.214477 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.214707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.214516 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.214707 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:24.214641 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-serving-cert: secret "maas-api-e2e-shared-a-w4-03a6e5-serving-cert" not found Nov 03 09:31:24.214832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.214646 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.214832 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:24.214709 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls podName:5e84ee51-3cbd-417a-a2a4-812e522c526a nodeName:}" failed. No retries permitted until 2026-11-03 09:31:24.714689119 +0000 UTC m=+3108.382729842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls") pod "maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" (UID: "5e84ee51-3cbd-417a-a2a4-812e522c526a") : secret "maas-api-e2e-shared-a-w4-03a6e5-serving-cert" not found Nov 03 09:31:24.214832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.214817 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk58b\" (UniqueName: \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.215298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.215276 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.217132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.217113 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.231350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.231329 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:24.275594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.275568 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk58b\" (UniqueName: \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.389544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.389509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:24.531413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.531380 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:24.722067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.721991 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.724480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.724452 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:24.953322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:24.953291 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:25.330633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:25.330605 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:31:25.332229 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:31:25.332204 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e84ee51_3cbd_417a_a2a4_812e522c526a.slice/crio-645f1ede1ebafb67b86f9de2094e15c1670bfb357a7dc5a8600cdcb6a288e5a5 WatchSource:0}: Error finding container 645f1ede1ebafb67b86f9de2094e15c1670bfb357a7dc5a8600cdcb6a288e5a5: Status 404 returned error can't find the container with id 645f1ede1ebafb67b86f9de2094e15c1670bfb357a7dc5a8600cdcb6a288e5a5 Nov 03 09:31:25.813102 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:25.813069 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" event={"ID":"5e84ee51-3cbd-417a-a2a4-812e522c526a","Type":"ContainerStarted","Data":"645f1ede1ebafb67b86f9de2094e15c1670bfb357a7dc5a8600cdcb6a288e5a5"} Nov 03 09:31:26.820729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:26.820622 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" event={"ID":"5e84ee51-3cbd-417a-a2a4-812e522c526a","Type":"ContainerStarted","Data":"e0ddc53e98dbf6625180a942baf5480e9c449a34fcc2db557ba9e81309b841e4"} Nov 03 09:31:26.821156 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:26.820826 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:26.941285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:26.941229 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" podStartSLOduration=2.8987484119999998 podStartE2EDuration="3.941215408s" totalImagesPullingTime="1.042466996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:31:26.938654983 +0000 UTC m=+3110.606695721" watchObservedRunningTime="2026-11-03 09:31:26.941215408 +0000 UTC m=+3110.609256138" Nov 03 09:31:29.426710 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:29.426675 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:32.833413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:32.833378 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:31:34.406883 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:34.406849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:35.563408 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:35.563363 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:31:39.477263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:39.477234 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:44.156636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.156583 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:31:44.162658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.162636 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.219263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.219232 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:31:44.264193 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.264164 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.264388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.264202 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.264388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.264269 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.264478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.264390 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnzps\" (UniqueName: \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.266005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.265984 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-739a7f-serving-cert\"" Nov 03 09:31:44.343539 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.343503 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:44.365890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.365861 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.366073 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.366002 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnzps\" (UniqueName: \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.366214 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.366199 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.366285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.366243 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.366384 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:44.366368 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-serving-cert: secret "maas-api-e2e-extmod-a-739a7f-serving-cert" not found Nov 03 09:31:44.366446 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:31:44.366441 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls podName:3859e0f1-89aa-4f34-ba39-819456b68754 nodeName:}" failed. No retries permitted until 2026-11-03 09:31:44.866420797 +0000 UTC m=+3128.534461527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls") pod "maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" (UID: "3859e0f1-89aa-4f34-ba39-819456b68754") : secret "maas-api-e2e-extmod-a-739a7f-serving-cert" not found Nov 03 09:31:44.366548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.366530 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.368736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.368714 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.425601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.425483 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnzps\" (UniqueName: \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.508294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.508256 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:44.634848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.634812 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:44.873708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.873676 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:44.876299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:44.876269 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") pod \"maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:45.076056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:45.076026 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:45.455201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:45.455169 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:31:45.456060 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:31:45.456034 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3859e0f1_89aa_4f34_ba39_819456b68754.slice/crio-74472780d230d6e3fa5b041fd39c072a00f1237d7d297a1fcb32c23931e9b805 WatchSource:0}: Error finding container 74472780d230d6e3fa5b041fd39c072a00f1237d7d297a1fcb32c23931e9b805: Status 404 returned error can't find the container with id 74472780d230d6e3fa5b041fd39c072a00f1237d7d297a1fcb32c23931e9b805 Nov 03 09:31:45.937933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:45.937890 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" event={"ID":"3859e0f1-89aa-4f34-ba39-819456b68754","Type":"ContainerStarted","Data":"74472780d230d6e3fa5b041fd39c072a00f1237d7d297a1fcb32c23931e9b805"} Nov 03 09:31:46.945321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:46.945276 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" event={"ID":"3859e0f1-89aa-4f34-ba39-819456b68754","Type":"ContainerStarted","Data":"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d"} Nov 03 09:31:46.945798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:46.945343 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:47.067698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:47.067647 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" podStartSLOduration=2.300992097 podStartE2EDuration="3.067634008s" totalImagesPullingTime="766.641911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:31:47.065419785 +0000 UTC m=+3130.733460525" watchObservedRunningTime="2026-11-03 09:31:47.067634008 +0000 UTC m=+3130.735674746" Nov 03 09:31:49.553081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:49.553041 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:52.956966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:52.956938 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:31:54.632846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:54.632816 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:56.667688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:56.667658 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:59.403417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:59.403379 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:59.543933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:59.543891 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:59.690189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:59.690065 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:31:59.818162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:31:59.818102 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:00.351326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:00.351269 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:01.002076 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:01.002017 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:01.907361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:01.907325 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:02.061934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.061899 4543 generic.go:356] "Generic (PLEG): container finished" podID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerID="5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5" exitCode=0 Nov 03 09:32:02.062433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.061982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerDied","Data":"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5"} Nov 03 09:32:02.062524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.062507 4543 scope.go:122] "RemoveContainer" containerID="5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5" Nov 03 09:32:02.064227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.064198 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:02.064339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.064270 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:02.064339 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.064309 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:02.238552 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:02.238458 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:03.069797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:03.069763 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerStarted","Data":"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554"} Nov 03 09:32:03.113227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:03.113191 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:32:03.113514 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:03.113482 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:03.113634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:03.113535 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:04.113394 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.113363 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:04.113795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.113430 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:04.506653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.506549 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:32:04.511714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.511689 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.560736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.560702 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:32:04.610449 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.610410 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f01a6d-serving-cert\"" Nov 03 09:32:04.613478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.613441 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.613478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.613480 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.613721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.613503 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.613721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.613682 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlx2\" (UniqueName: \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.710021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.709977 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:04.715568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.715541 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.715747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.715587 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.715747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.715616 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.715747 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:32:04.715663 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-serving-cert: secret "maas-api-e2e-extmod-a-f01a6d-serving-cert" not found Nov 03 09:32:04.715747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.715718 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlx2\" (UniqueName: \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.715747 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:32:04.715743 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls podName:5395bc94-09f5-4992-9fb4-05ed3ad9fb20 nodeName:}" failed. No retries permitted until 2026-11-03 09:32:05.215722509 +0000 UTC m=+3148.883763242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls") pod "maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" (UID: "5395bc94-09f5-4992-9fb4-05ed3ad9fb20") : secret "maas-api-e2e-extmod-a-f01a6d-serving-cert" not found Nov 03 09:32:04.716303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.716280 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.718428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.718405 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.774932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.774860 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlx2\" (UniqueName: \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:04.907625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:04.907590 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:05.037095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.036988 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:05.114062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.114029 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:05.114489 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.114099 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:05.223279 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.223245 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:05.225985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.225957 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:05.425584 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.425545 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:05.567279 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:32:05.567242 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:32:05.824344 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:05.824313 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:32:05.825085 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:32:05.825062 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5395bc94_09f5_4992_9fb4_05ed3ad9fb20.slice/crio-6611d2f4c7fe991dcfc009c3baf7b3231120c3b2d9ebcde274367f540b401b83 WatchSource:0}: Error finding container 6611d2f4c7fe991dcfc009c3baf7b3231120c3b2d9ebcde274367f540b401b83: Status 404 returned error can't find the container with id 6611d2f4c7fe991dcfc009c3baf7b3231120c3b2d9ebcde274367f540b401b83 Nov 03 09:32:06.110932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:06.110897 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" event={"ID":"5395bc94-09f5-4992-9fb4-05ed3ad9fb20","Type":"ContainerStarted","Data":"6611d2f4c7fe991dcfc009c3baf7b3231120c3b2d9ebcde274367f540b401b83"} Nov 03 09:32:06.113373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:06.113339 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:06.113507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:06.113385 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:07.110466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.110379 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:07.113792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.113767 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:07.113940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.113812 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:07.117984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.117959 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" event={"ID":"5395bc94-09f5-4992-9fb4-05ed3ad9fb20","Type":"ContainerStarted","Data":"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1"} Nov 03 09:32:07.118110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.118094 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:07.243747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.243676 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" podStartSLOduration=2.320753145 podStartE2EDuration="3.24365728s" totalImagesPullingTime="922.904135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:07.241339497 +0000 UTC m=+3150.909380284" watchObservedRunningTime="2026-11-03 09:32:07.24365728 +0000 UTC m=+3150.911698020" Nov 03 09:32:07.307365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.307259 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:07.475933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:07.475897 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:08.114027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:08.113993 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 09:32:08.114466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:08.114055 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 09:32:08.322903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:08.322854 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:09.118724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:09.118693 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:32:09.119243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:09.119108 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:32:09.119927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:09.119911 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:32:10.281528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:10.281487 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:13.131111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:13.131082 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:32:14.639341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:14.639281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:14.817517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:14.817474 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:15.719033 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:15.718997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:21.056321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:21.056288 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:24.446358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:24.446306 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:32:24.446781 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:24.446429 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:32:25.295090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.295000 4543 generic.go:356] "Generic (PLEG): container finished" podID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerID="970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5" exitCode=0 Nov 03 09:32:25.295090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.295079 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerDied","Data":"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5"} Nov 03 09:32:25.295589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.295569 4543 scope.go:122] "RemoveContainer" containerID="970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5" Nov 03 09:32:25.296996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.296975 4543 generic.go:356] "Generic (PLEG): container finished" podID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerID="20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6" exitCode=0 Nov 03 09:32:25.297106 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.297054 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerDied","Data":"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6"} Nov 03 09:32:25.297412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.297388 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:25.297459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.297435 4543 scope.go:122] "RemoveContainer" containerID="20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6" Nov 03 09:32:25.297493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.297458 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:25.297530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.297491 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:25.299074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.299045 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:25.299180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.299124 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:25.299223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:25.299178 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:26.024257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.024219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:26.319673 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.319574 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerStarted","Data":"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903"} Nov 03 09:32:26.323762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.323728 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerStarted","Data":"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f"} Nov 03 09:32:26.569639 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.569563 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:32:26.569870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.569784 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:26.569870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:26.569830 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:27.236551 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:27.236515 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:32:27.236968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:27.236622 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:32:27.236968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:27.236658 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 09:32:27.570271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:27.570191 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:27.570271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:27.570254 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:28.236563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:28.236530 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:32:28.236996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:28.236601 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 09:32:28.570916 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:28.570821 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:28.570916 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:28.570894 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:29.236374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:29.236328 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:32:29.236559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:29.236390 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 09:32:29.570428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:29.570343 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:29.570428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:29.570395 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:30.236584 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:30.236551 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:32:30.236790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:30.236620 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 09:32:30.569768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:30.569687 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:30.569768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:30.569744 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:31.161251 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:31.161217 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:31.237201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:31.236493 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Nov 03 09:32:31.237201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:31.236558 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Nov 03 09:32:31.570436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:31.570349 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 09:32:31.570436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:31.570409 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 09:32:32.240630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.240590 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:32:32.241085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.240938 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:32:32.241757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.241740 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:32:32.575264 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.575190 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:32:32.575617 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.575597 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:32:32.576821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:32.576798 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:32:35.561506 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:32:35.561467 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:32:36.141315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:36.141275 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:41.137755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:41.137723 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:46.128881 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:46.128848 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:52.056476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:52.056389 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:56.245830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:56.245799 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:57.014890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:57.014853 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:57.796932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:57.796892 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:32:58.546831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.546791 4543 generic.go:356] "Generic (PLEG): container finished" podID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerID="b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06" exitCode=0 Nov 03 09:32:58.547110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.546868 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerDied","Data":"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06"} Nov 03 09:32:58.547418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.547401 4543 scope.go:122] "RemoveContainer" containerID="b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06" Nov 03 09:32:58.549113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.549082 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:58.549260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.549188 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:58.549260 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:58.549239 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:32:59.554874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:32:59.554842 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerStarted","Data":"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df"} Nov 03 09:33:00.129295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:00.129264 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:33:00.129500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:00.129433 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:00.129569 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:00.129504 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:01.129475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:01.129439 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:01.129875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:01.129493 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:01.354516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:01.354484 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:02.128685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:02.128649 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:02.128876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:02.128713 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:03.128583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:03.128546 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:03.129049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:03.128627 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:04.129073 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:04.129034 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:04.129597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:04.129087 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:05.129496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:05.129456 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 09:33:05.129930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:05.129535 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 09:33:05.563672 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:33:05.563564 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:33:05.879807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:05.879773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:06.134396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:06.134309 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:33:06.134789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:06.134686 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:33:06.135316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:06.135297 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:33:06.358303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:06.358270 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:06.647410 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:06.647377 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:07.197929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:07.197896 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:07.407055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:07.407023 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:08.665103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:08.665073 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:11.307042 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:11.306974 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:11.489737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:11.489676 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:11.622856 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:11.622805 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:12.081970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:12.081890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:16.697155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:16.697114 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:16.878191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:16.878129 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:17.248885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:17.248850 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:22.302638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:22.302597 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:25.951361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:25.951328 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:27.560099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:27.560063 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:32.547340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:32.547298 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:32.687086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:32.687055 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:33.658464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:33.658420 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:33.787913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:33.787877 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:35.560778 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:33:35.560741 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:33:37.715480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.715447 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:33:37.718309 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.718292 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.795836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.795802 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.796021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.795929 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtx9\" (UniqueName: \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.796021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.795988 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.796195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.796116 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.823361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.823339 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-85fdd1-serving-cert\"" Nov 03 09:33:37.826677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.826652 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:33:37.897755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.897719 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.897952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.897821 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtx9\" (UniqueName: \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.897952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.897866 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.897952 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:33:37.897892 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-serving-cert: secret "maas-api-e2e-autores-85fdd1-serving-cert" not found Nov 03 09:33:37.897952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.897920 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.898235 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:33:37.897974 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls podName:9a5e817c-6865-4bd4-b2ab-da9ea32bc07f nodeName:}" failed. No retries permitted until 2026-11-03 09:33:38.397953048 +0000 UTC m=+3242.065993786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls") pod "maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" (UID: "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f") : secret "maas-api-e2e-autores-85fdd1-serving-cert" not found Nov 03 09:33:37.898653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.898632 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.900410 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.900386 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:37.902613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.902588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:37.969099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:37.968973 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtx9\" (UniqueName: \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:38.404780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:38.404748 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:38.407293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:38.407269 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") pod \"maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:38.629982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:38.629950 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:39.008497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:39.007568 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:33:39.096325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:39.096295 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:39.402284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:39.402251 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:39.838525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:39.838439 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" event={"ID":"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f","Type":"ContainerStarted","Data":"f579345fa23cd58d1de8fb7a8049fdb7abe20bc6d907b21f5bd6659f53d34c9f"} Nov 03 09:33:40.091553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:40.091463 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:40.846227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:40.846196 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" event={"ID":"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f","Type":"ContainerStarted","Data":"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d"} Nov 03 09:33:40.846423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:40.846259 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:43.544722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:43.544683 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:44.283855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:44.283815 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:46.691490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:46.691437 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:46.858420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:46.858390 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:33:46.981318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:46.981202 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" podStartSLOduration=9.128881649 podStartE2EDuration="9.981183501s" totalImagesPullingTime="852.301852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:33:40.97556335 +0000 UTC m=+3244.643604088" watchObservedRunningTime="2026-11-03 09:33:46.981183501 +0000 UTC m=+3250.649224241" Nov 03 09:33:47.022032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:47.021992 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:47.156857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:47.156802 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:48.714598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:48.714562 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:50.111584 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:50.111547 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:50.240534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:50.240507 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:51.574988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:51.574952 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:53.001608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:53.001575 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:54.035391 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:54.035359 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:54.717243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:54.717209 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:57.356202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:57.356168 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:57.643940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:57.643908 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:33:59.035660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:33:59.035623 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:04.181279 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:04.181245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:05.561209 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:34:05.561167 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:34:09.009901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:09.009867 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:09.298677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:09.298588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:14.368132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:14.368097 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:19.342925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:19.342848 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:24.360675 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:24.360599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:29.354353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:29.354316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:34.436590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:34.436557 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:35.562022 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:34:35.561985 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:34:36.889708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:36.889684 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:34:36.956510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:36.956485 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:34:39.442074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:39.442044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:44.428274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:44.428239 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:49.381737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:49.381699 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:49.527608 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:49.527576 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:49.655737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:49.655641 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:54.555885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:54.555847 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:57.704975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:57.704917 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:58.487816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:58.487771 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:34:59.850504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:34:59.850466 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:04.977311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:04.977264 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:05.561545 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:35:05.561511 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:35:09.335524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:09.335493 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:09.470890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:09.470853 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:11.238960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:11.238928 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:11.989000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:11.988962 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:12.613177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:12.613131 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:35:12.613674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:12.613363 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" containerID="cri-o://9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903" gracePeriod=30 Nov 03 09:35:12.659360 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:12.659322 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:13.572412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:13.572381 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:13.707874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:13.707842 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:16.772248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:16.772210 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:17.578189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:17.578130 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:35:17.578363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:17.578222 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:35:17.894034 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:17.894006 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:35:18.085396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085364 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085437 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085505 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085533 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085594 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085666 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085681 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:35:18.085808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085702 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.085970 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085896 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.086026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.085970 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\") pod \"7f4399c8-4654-49b1-82cd-a6d22f04d919\" (UID: \"7f4399c8-4654-49b1-82cd-a6d22f04d919\") " Nov 03 09:35:18.086081 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086038 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:35:18.086419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086369 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:35:18.086529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086427 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:35:18.086529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086475 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:35:18.087006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086854 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.087006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086882 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.087006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086898 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4399c8-4654-49b1-82cd-a6d22f04d919-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.087006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086912 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.087006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.086926 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.088418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.088387 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:35:18.088637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.088593 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "kube-api-access-67z5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:35:18.088944 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.088923 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:35:18.089000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.088923 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo" pod "7f4399c8-4654-49b1-82cd-a6d22f04d919" (UID: "7f4399c8-4654-49b1-82cd-a6d22f04d919"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:35:18.188625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.188537 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67z5s\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-kube-api-access-67z5s\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.188625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.188569 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.188625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.188578 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.188625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.188587 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4399c8-4654-49b1-82cd-a6d22f04d919-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:35:18.454110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.454027 4543 generic.go:356] "Generic (PLEG): container finished" podID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerID="9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903" exitCode=0 Nov 03 09:35:18.454292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.454154 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" Nov 03 09:35:18.456026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.455988 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerDied","Data":"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903"} Nov 03 09:35:18.456181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.456039 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r" event={"ID":"7f4399c8-4654-49b1-82cd-a6d22f04d919","Type":"ContainerDied","Data":"950959ba6d5df62d6e01090a19bf7eef3671da3ba4e2c8dd8374cd6e90bba74b"} Nov 03 09:35:18.456181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.456119 4543 scope.go:122] "RemoveContainer" containerID="9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903" Nov 03 09:35:18.468764 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.468743 4543 scope.go:122] "RemoveContainer" containerID="970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5" Nov 03 09:35:18.478249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.478222 4543 scope.go:122] "RemoveContainer" containerID="9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903" Nov 03 09:35:18.478520 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:35:18.478500 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903\": container with ID starting with 9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903 not found: ID does not exist" containerID="9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903" Nov 03 09:35:18.478597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.478537 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903"} err="failed to get container status \"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903\": rpc error: code = NotFound desc = could not find container \"9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903\": container with ID starting with 9f532a440845fc438904b3ac1d17d45e2e4260e588f87d0c5d8030ed2a499903 not found: ID does not exist" Nov 03 09:35:18.478597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.478564 4543 scope.go:122] "RemoveContainer" containerID="970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5" Nov 03 09:35:18.478825 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:35:18.478808 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5\": container with ID starting with 970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5 not found: ID does not exist" containerID="970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5" Nov 03 09:35:18.478866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.478832 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5"} err="failed to get container status \"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5\": rpc error: code = NotFound desc = could not find container \"970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5\": container with ID starting with 970388afccb2f6500e0149b95f140ee62f3f5fceabc2207d1e539a0a06cf04c5 not found: ID does not exist" Nov 03 09:35:18.627609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.627573 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:35:18.637380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:18.637353 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f66c6c-openshift-default-6f87df9549-b9q8r"] Nov 03 09:35:20.240815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:20.240766 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:20.452266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:20.452234 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" path="/var/lib/kubelet/pods/7f4399c8-4654-49b1-82cd-a6d22f04d919/volumes" Nov 03 09:35:22.305387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:22.305325 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:26.093974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:26.093940 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:26.229562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:26.229518 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:26.857747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:26.857719 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:27.563993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:27.563943 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:29.739337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:29.739294 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:29.875923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:29.875889 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:32.737733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:32.737694 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:33.174762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:33.174727 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:33.389748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:33.389717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:35.562274 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:35:35.562236 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:35:37.746499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:37.746463 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:37.958047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:37.958009 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:38.096284 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:38.096192 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:42.743419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:42.743378 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:47.744443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:47.744367 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:52.732438 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:52.732397 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:57.740867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:57.740824 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:58.046316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:58.046233 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:35:58.861632 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:35:58.861597 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:02.730611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:02.730580 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:05.561518 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:05.561480 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:36:07.742503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:07.742467 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:12.760345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:12.760308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:13.576241 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:13.576205 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:14.601510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:14.601473 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:17.899558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:17.899517 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:19.711230 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:19.711188 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:21.736263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:21.736222 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:21.959414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:21.959379 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:22.105737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:22.105653 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:23.121455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:23.121421 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:25.799908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:25.799880 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:27.999057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:27.999028 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:36:27.999503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:27.999271 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" podUID="3859e0f1-89aa-4f34-ba39-819456b68754" containerName="maas-api" containerID="cri-o://ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d" gracePeriod=30 Nov 03 09:36:28.004901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.004675 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:28.143794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.143757 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:28.280401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.280373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:28.283817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.283797 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:36:28.363950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.363922 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") pod \"3859e0f1-89aa-4f34-ba39-819456b68754\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " Nov 03 09:36:28.364124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.363994 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\" (UniqueName: \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\") pod \"3859e0f1-89aa-4f34-ba39-819456b68754\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " Nov 03 09:36:28.364124 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.364112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\") pod \"3859e0f1-89aa-4f34-ba39-819456b68754\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " Nov 03 09:36:28.364310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.364210 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\") pod \"3859e0f1-89aa-4f34-ba39-819456b68754\" (UID: \"3859e0f1-89aa-4f34-ba39-819456b68754\") " Nov 03 09:36:28.364493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.364470 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca" pod "3859e0f1-89aa-4f34-ba39-819456b68754" (UID: "3859e0f1-89aa-4f34-ba39-819456b68754"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:28.364763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.364698 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3859e0f1-89aa-4f34-ba39-819456b68754-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:28.366290 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.366264 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls" pod "3859e0f1-89aa-4f34-ba39-819456b68754" (UID: "3859e0f1-89aa-4f34-ba39-819456b68754"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:28.366390 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.366327 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs" pod "3859e0f1-89aa-4f34-ba39-819456b68754" (UID: "3859e0f1-89aa-4f34-ba39-819456b68754"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:28.366390 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.366353 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps" pod "3859e0f1-89aa-4f34-ba39-819456b68754" (UID: "3859e0f1-89aa-4f34-ba39-819456b68754"). InnerVolumeSpecName "kube-api-access-rnzps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:28.467652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.467540 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:28.467652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.467572 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnzps\" (UniqueName: \"kubernetes.io/projected/3859e0f1-89aa-4f34-ba39-819456b68754-kube-api-access-rnzps\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:28.467652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.467587 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3859e0f1-89aa-4f34-ba39-819456b68754-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:28.938623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.938586 4543 generic.go:356] "Generic (PLEG): container finished" podID="3859e0f1-89aa-4f34-ba39-819456b68754" containerID="ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d" exitCode=0 Nov 03 09:36:28.938837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.938631 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" event={"ID":"3859e0f1-89aa-4f34-ba39-819456b68754","Type":"ContainerDied","Data":"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d"} Nov 03 09:36:28.938837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.938641 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" Nov 03 09:36:28.938837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.938686 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq" event={"ID":"3859e0f1-89aa-4f34-ba39-819456b68754","Type":"ContainerDied","Data":"74472780d230d6e3fa5b041fd39c072a00f1237d7d297a1fcb32c23931e9b805"} Nov 03 09:36:28.938837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.938716 4543 scope.go:122] "RemoveContainer" containerID="ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d" Nov 03 09:36:28.948696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.948675 4543 scope.go:122] "RemoveContainer" containerID="ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d" Nov 03 09:36:28.948996 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:28.948975 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d\": container with ID starting with ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d not found: ID does not exist" containerID="ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d" Nov 03 09:36:28.949063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:28.949009 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d"} err="failed to get container status \"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d\": rpc error: code = NotFound desc = could not find container \"ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d\": container with ID starting with ceb56a26b540bf7a60bb0e05903ea21195f5f5771ffd235ed768c38d4f148c8d not found: ID does not exist" Nov 03 09:36:29.140388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:29.123188 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:36:29.144614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:29.144579 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-739a7f-6d9476494d-468sq"] Nov 03 09:36:30.044421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.044387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:30.176001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.175964 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:30.451949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.451916 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3859e0f1-89aa-4f34-ba39-819456b68754" path="/var/lib/kubelet/pods/3859e0f1-89aa-4f34-ba39-819456b68754/volumes" Nov 03 09:36:30.584997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.584967 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:30.598750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.598723 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:36:30.598936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:30.598916 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" containerID="cri-o://0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f" gracePeriod=30 Nov 03 09:36:32.243868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:32.243822 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:36:32.244291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:32.243894 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:36:33.640414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:33.640379 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:33.776357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:33.776324 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:35.560784 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:35.560744 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:36:35.680896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.680866 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:35.864576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.864553 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:36:35.988494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.988456 4543 generic.go:356] "Generic (PLEG): container finished" podID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerID="0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f" exitCode=0 Nov 03 09:36:35.988677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.988509 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" Nov 03 09:36:35.988677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.988513 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerDied","Data":"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f"} Nov 03 09:36:35.988677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.988568 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw" event={"ID":"8a8f298c-75bd-4615-9aa5-49fe66f7b97f","Type":"ContainerDied","Data":"e7103c0342f4bf6391e1cebb51f9e81152762cfecffd7a43b38e9b97fc995450"} Nov 03 09:36:35.988677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.988600 4543 scope.go:122] "RemoveContainer" containerID="0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f" Nov 03 09:36:35.990867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.990844 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.990983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.990921 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.990983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.990944 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.990983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.990974 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.991188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991033 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.991188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991070 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.991188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991166 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:35.991367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991281 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:35.991367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991295 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:35.991367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991284 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.991537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991428 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.991537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991456 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:35.991537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991462 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:35.991537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.991467 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\") pod \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\" (UID: \"8a8f298c-75bd-4615-9aa5-49fe66f7b97f\") " Nov 03 09:36:35.992254 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.992228 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:35.992380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.992258 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:35.992380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.992274 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:35.992380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.992287 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:35.992380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.992303 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:35.993562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.993470 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "kube-api-access-xh7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:35.993562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.993542 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:36:35.993772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.993753 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:35.994073 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:35.994054 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token" pod "8a8f298c-75bd-4615-9aa5-49fe66f7b97f" (UID: "8a8f298c-75bd-4615-9aa5-49fe66f7b97f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:36.019800 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.019781 4543 scope.go:122] "RemoveContainer" containerID="20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6" Nov 03 09:36:36.031989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.031961 4543 scope.go:122] "RemoveContainer" containerID="0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f" Nov 03 09:36:36.032336 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:36.032312 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f\": container with ID starting with 0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f not found: ID does not exist" containerID="0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f" Nov 03 09:36:36.032417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.032346 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f"} err="failed to get container status \"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f\": rpc error: code = NotFound desc = could not find container \"0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f\": container with ID starting with 0829ee2a50cdeaf3775546bc6ddc6e0315e8fc81db05ed603430c07d14a0d54f not found: ID does not exist" Nov 03 09:36:36.032417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.032367 4543 scope.go:122] "RemoveContainer" containerID="20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6" Nov 03 09:36:36.032611 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:36.032597 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6\": container with ID starting with 20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6 not found: ID does not exist" containerID="20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6" Nov 03 09:36:36.032662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.032615 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6"} err="failed to get container status \"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6\": rpc error: code = NotFound desc = could not find container \"20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6\": container with ID starting with 20fb5917e72dce68d439605329671db9400bee06cd29e667ab33c03939ed89d6 not found: ID does not exist" Nov 03 09:36:36.093971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.093888 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:36.093971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.093918 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7rm\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-kube-api-access-xh7rm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:36.093971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.093931 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:36.093971 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.093939 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a8f298c-75bd-4615-9aa5-49fe66f7b97f-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:36.415152 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.415106 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:36:36.465606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:36.465574 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-739a7f-openshift-default-5d6b59bf69-td8qw"] Nov 03 09:36:38.451677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:38.451642 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" path="/var/lib/kubelet/pods/8a8f298c-75bd-4615-9aa5-49fe66f7b97f/volumes" Nov 03 09:36:39.688501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:39.688457 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:39.830682 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:39.830644 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:40.098661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.098566 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:36:40.098894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.098868 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" podUID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" containerName="maas-api" containerID="cri-o://b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d" gracePeriod=30 Nov 03 09:36:40.190294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.190258 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:40.381882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.381859 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:36:40.448042 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\") pod \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " Nov 03 09:36:40.448203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448050 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\" (UniqueName: \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\") pod \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " Nov 03 09:36:40.448271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448234 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\") pod \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " Nov 03 09:36:40.448394 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448379 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") pod \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\" (UID: \"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f\") " Nov 03 09:36:40.448640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448604 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca" pod "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" (UID: "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:40.448897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.448857 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:40.450452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.450431 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9" pod "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" (UID: "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f"). InnerVolumeSpecName "kube-api-access-tqtx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:40.450452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.450438 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs" pod "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" (UID: "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:40.450862 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.450833 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls" pod "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" (UID: "9a5e817c-6865-4bd4-b2ab-da9ea32bc07f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:40.549963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.549924 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:40.549963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.549961 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:40.549963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:40.549971 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtx9\" (UniqueName: \"kubernetes.io/projected/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f-kube-api-access-tqtx9\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:41.020662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.020617 4543 generic.go:356] "Generic (PLEG): container finished" podID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" containerID="b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d" exitCode=0 Nov 03 09:36:41.021173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.020669 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" Nov 03 09:36:41.021173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.020702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" event={"ID":"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f","Type":"ContainerDied","Data":"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d"} Nov 03 09:36:41.021173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.020759 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj" event={"ID":"9a5e817c-6865-4bd4-b2ab-da9ea32bc07f","Type":"ContainerDied","Data":"f579345fa23cd58d1de8fb7a8049fdb7abe20bc6d907b21f5bd6659f53d34c9f"} Nov 03 09:36:41.021173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.020787 4543 scope.go:122] "RemoveContainer" containerID="b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d" Nov 03 09:36:41.030836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.030814 4543 scope.go:122] "RemoveContainer" containerID="b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d" Nov 03 09:36:41.031133 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:41.031108 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d\": container with ID starting with b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d not found: ID does not exist" containerID="b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d" Nov 03 09:36:41.031133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.031169 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d"} err="failed to get container status \"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d\": rpc error: code = NotFound desc = could not find container \"b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d\": container with ID starting with b7496bd4d4c6f9f6a1ba383e5674c7e9a9c47071007db0d4c3349b1b7d6e6b5d not found: ID does not exist" Nov 03 09:36:41.190458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.190428 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:36:41.240874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.240843 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-85fdd1-57cb4f5799-w4tpj"] Nov 03 09:36:41.311750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.311678 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:36:41.311920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.311879 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" podUID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" containerName="maas-api" containerID="cri-o://0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1" gracePeriod=30 Nov 03 09:36:41.401538 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.401483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:41.585469 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.585401 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:36:41.663334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.663303 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") pod \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " Nov 03 09:36:41.663529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.663349 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\" (UniqueName: \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\") pod \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " Nov 03 09:36:41.663529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.663407 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\") pod \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " Nov 03 09:36:41.663529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.663425 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\") pod \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\" (UID: \"5395bc94-09f5-4992-9fb4-05ed3ad9fb20\") " Nov 03 09:36:41.663816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.663786 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca" pod "5395bc94-09f5-4992-9fb4-05ed3ad9fb20" (UID: "5395bc94-09f5-4992-9fb4-05ed3ad9fb20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:41.665712 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.665686 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs" pod "5395bc94-09f5-4992-9fb4-05ed3ad9fb20" (UID: "5395bc94-09f5-4992-9fb4-05ed3ad9fb20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:41.665810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.665708 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls" pod "5395bc94-09f5-4992-9fb4-05ed3ad9fb20" (UID: "5395bc94-09f5-4992-9fb4-05ed3ad9fb20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:36:41.665908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.665888 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2" pod "5395bc94-09f5-4992-9fb4-05ed3ad9fb20" (UID: "5395bc94-09f5-4992-9fb4-05ed3ad9fb20"). InnerVolumeSpecName "kube-api-access-jmlx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:41.765444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.765408 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:41.765444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.765437 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmlx2\" (UniqueName: \"kubernetes.io/projected/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-kube-api-access-jmlx2\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:41.765444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.765447 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:41.765692 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:41.765457 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5395bc94-09f5-4992-9fb4-05ed3ad9fb20-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:42.029218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.029182 4543 generic.go:356] "Generic (PLEG): container finished" podID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" containerID="0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1" exitCode=0 Nov 03 09:36:42.029218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.029218 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" event={"ID":"5395bc94-09f5-4992-9fb4-05ed3ad9fb20","Type":"ContainerDied","Data":"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1"} Nov 03 09:36:42.029709 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.029234 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" Nov 03 09:36:42.029709 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.029257 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7" event={"ID":"5395bc94-09f5-4992-9fb4-05ed3ad9fb20","Type":"ContainerDied","Data":"6611d2f4c7fe991dcfc009c3baf7b3231120c3b2d9ebcde274367f540b401b83"} Nov 03 09:36:42.029709 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.029279 4543 scope.go:122] "RemoveContainer" containerID="0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1" Nov 03 09:36:42.039506 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.039473 4543 scope.go:122] "RemoveContainer" containerID="0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1" Nov 03 09:36:42.039793 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:42.039774 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1\": container with ID starting with 0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1 not found: ID does not exist" containerID="0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1" Nov 03 09:36:42.039866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.039803 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1"} err="failed to get container status \"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1\": rpc error: code = NotFound desc = could not find container \"0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1\": container with ID starting with 0f40f79b594aaa4969e63bb67eaa52944204fac94fe4c02425e6df39b7a66ef1 not found: ID does not exist" Nov 03 09:36:42.201424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.201393 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:36:42.251833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.251791 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f01a6d-57d6f98f99-xdgl7"] Nov 03 09:36:42.451918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.451874 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" path="/var/lib/kubelet/pods/5395bc94-09f5-4992-9fb4-05ed3ad9fb20/volumes" Nov 03 09:36:42.452440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:42.452418 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" path="/var/lib/kubelet/pods/9a5e817c-6865-4bd4-b2ab-da9ea32bc07f/volumes" Nov 03 09:36:43.213774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:43.213736 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:43.347057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:43.347023 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:43.625877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:43.625851 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:43.631464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:43.631444 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:36:43.631647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:43.631627 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" containerID="cri-o://8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df" gracePeriod=30 Nov 03 09:36:46.625803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:46.625768 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:46.755937 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:46.755902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:48.662827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.662778 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:48.890533 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.890506 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:36:48.995766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.995689 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.995766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.995728 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.995766 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.995752 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.996063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.995797 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.996063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.995874 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.996063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996029 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:48.996291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996098 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.996291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996222 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.997085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996268 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.997085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996350 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:48.997085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996658 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:48.997085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996727 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:48.997085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.996840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\") pod \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\" (UID: \"d49d17b5-47ee-4910-81f0-8b8ac0d56cad\") " Nov 03 09:36:48.997490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.997279 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:36:48.998017 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.997989 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:48.998222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.998020 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:48.998222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.998036 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:48.998222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.998052 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:48.998222 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.998065 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:48.998809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.998768 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "kube-api-access-l59c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:48.999224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.999197 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:36:48.999304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.999195 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:36:48.999304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:48.999224 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy" pod "d49d17b5-47ee-4910-81f0-8b8ac0d56cad" (UID: "d49d17b5-47ee-4910-81f0-8b8ac0d56cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:36:49.077032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.076996 4543 generic.go:356] "Generic (PLEG): container finished" podID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerID="8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df" exitCode=0 Nov 03 09:36:49.077300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.077065 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" Nov 03 09:36:49.077300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.077087 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerDied","Data":"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df"} Nov 03 09:36:49.077300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.077193 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx" event={"ID":"d49d17b5-47ee-4910-81f0-8b8ac0d56cad","Type":"ContainerDied","Data":"136de4bc053c5536d691f6428fd9aaec103ce848315c5297dee85c7d055cd3e0"} Nov 03 09:36:49.077300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.077228 4543 scope.go:122] "RemoveContainer" containerID="8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df" Nov 03 09:36:49.094918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.094894 4543 scope.go:122] "RemoveContainer" containerID="b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06" Nov 03 09:36:49.099732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.099701 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:49.099732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.099734 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:49.099928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.099750 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:49.099928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.099765 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l59c9\" (UniqueName: \"kubernetes.io/projected/d49d17b5-47ee-4910-81f0-8b8ac0d56cad-kube-api-access-l59c9\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:36:49.105913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.105891 4543 scope.go:122] "RemoveContainer" containerID="8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df" Nov 03 09:36:49.106328 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:49.106297 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df\": container with ID starting with 8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df not found: ID does not exist" containerID="8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df" Nov 03 09:36:49.106434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.106342 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df"} err="failed to get container status \"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df\": rpc error: code = NotFound desc = could not find container \"8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df\": container with ID starting with 8e164d31f967280fbf946a2d3bf34cca825d1384b19ac9623b1e57eacbaf28df not found: ID does not exist" Nov 03 09:36:49.106434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.106367 4543 scope.go:122] "RemoveContainer" containerID="b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06" Nov 03 09:36:49.106717 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:36:49.106693 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06\": container with ID starting with b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06 not found: ID does not exist" containerID="b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06" Nov 03 09:36:49.106791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.106721 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06"} err="failed to get container status \"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06\": rpc error: code = NotFound desc = could not find container \"b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06\": container with ID starting with b85edfe242cba86f9245919850259c22e20dd535d799311921cb6c2d06347b06 not found: ID does not exist" Nov 03 09:36:49.260174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.260069 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:36:49.262579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:49.262554 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-r6rvx"] Nov 03 09:36:50.453395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.453361 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" path="/var/lib/kubelet/pods/d49d17b5-47ee-4910-81f0-8b8ac0d56cad/volumes" Nov 03 09:36:50.484723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.484677 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:50.623712 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.623621 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:50.761715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.761634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:50.988601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.988560 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:36:50.989110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989096 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989110 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989125 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989131 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989156 4543 cpu_manager.go:414] "containerMap: removing container" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989160 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989167 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989172 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989182 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3859e0f1-89aa-4f34-ba39-819456b68754" containerName="maas-api" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989187 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3859e0f1-89aa-4f34-ba39-819456b68754" containerName="maas-api" Nov 03 09:36:50.989191 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989195 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989201 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989212 4543 cpu_manager.go:414] "containerMap: removing container" podUID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989217 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989235 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989240 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989312 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989319 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989326 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4399c8-4654-49b1-82cd-a6d22f04d919" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989331 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a8f298c-75bd-4615-9aa5-49fe66f7b97f" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989336 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3859e0f1-89aa-4f34-ba39-819456b68754" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989343 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989348 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5e817c-6865-4bd4-b2ab-da9ea32bc07f" containerName="maas-api" Nov 03 09:36:50.989520 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.989356 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5395bc94-09f5-4992-9fb4-05ed3ad9fb20" containerName="maas-api" Nov 03 09:36:50.994116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:50.994099 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.050207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.050122 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:36:51.097751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.097712 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-546e9f5ec78c\"" Nov 03 09:36:51.097913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.097812 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-05110bbd79d3\"" Nov 03 09:36:51.097913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.097858 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 09:36:51.097913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.097829 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b1c3ed9d233a\"" Nov 03 09:36:51.097913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.097876 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 09:36:51.098359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.098344 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gkdr4\"" Nov 03 09:36:51.098359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.098356 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 09:36:51.098471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.098364 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 09:36:51.124688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.124658 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.124809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.124691 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.124809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.124714 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprqw\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.124809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.124780 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.226288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226253 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.226288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226288 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.226535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226310 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprqw\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.226535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226339 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.226922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226900 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.227013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.226949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.229021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.229000 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.285618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.285590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprqw\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\") pod \"payload-processing-external-model-5c4fddf7b5-kmvr8\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.307549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.307484 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:51.685476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.685447 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:36:51.686994 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:36:51.686972 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996396de_ccf4_44be_90c1_ac25611c3505.slice/crio-3ace3157f65770d2eb9c29bc653572f37e749f1c87c9cdc2d9ecc983d74cc95f WatchSource:0}: Error finding container 3ace3157f65770d2eb9c29bc653572f37e749f1c87c9cdc2d9ecc983d74cc95f: Status 404 returned error can't find the container with id 3ace3157f65770d2eb9c29bc653572f37e749f1c87c9cdc2d9ecc983d74cc95f Nov 03 09:36:51.810244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:51.810212 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:52.101406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:52.101326 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" event={"ID":"996396de-ccf4-44be-90c1-ac25611c3505","Type":"ContainerStarted","Data":"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069"} Nov 03 09:36:52.101406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:52.101383 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" event={"ID":"996396de-ccf4-44be-90c1-ac25611c3505","Type":"ContainerStarted","Data":"3ace3157f65770d2eb9c29bc653572f37e749f1c87c9cdc2d9ecc983d74cc95f"} Nov 03 09:36:52.101631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:52.101490 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:36:52.269951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:52.269883 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" podStartSLOduration=2.26985382 podStartE2EDuration="2.26985382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:36:52.268754033 +0000 UTC m=+3435.936794773" watchObservedRunningTime="2026-11-03 09:36:52.26985382 +0000 UTC m=+3435.937894560" Nov 03 09:36:52.320853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:52.320825 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:53.969827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:53.969798 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:54.119256 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.119217 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:54.636454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.636408 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:36:54.636997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.636983 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:54.637045 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.637000 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:54.637092 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.637084 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49d17b5-47ee-4910-81f0-8b8ac0d56cad" containerName="istio-proxy" Nov 03 09:36:54.638927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.638899 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:54.640795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.640781 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664403 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664486 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664620 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664601 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664646 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664798 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664841 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.664920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664884 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxrf\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.665018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.664925 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.733863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.733812 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-373116-openshift-default-dockercfg-l9cdt\"" Nov 03 09:36:54.751746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.751713 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:36:54.765948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.765914 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.765966 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766006 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxrf\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766042 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766107 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766188 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766238 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766285 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766324 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766491 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766752 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766806 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.766948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.766809 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.767213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.767186 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.769234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.769207 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.769739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.769718 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.777672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.777650 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:54.825417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.825380 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxrf\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.825586 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.825529 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\") pod \"e2e-extmod-a-373116-openshift-default-559449785-2jxqb\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:54.942668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.942175 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:54.954553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:54.954525 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:55.391031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:55.390994 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:36:55.429931 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:36:55.429903 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c46e78a_dc06_463a_8e74_7bd5f4cb0381.slice/crio-b54c40ffcf6e186b61d03c3ea5e793cdb1ff39d385c28c47a052fbc426ef4c38 WatchSource:0}: Error finding container b54c40ffcf6e186b61d03c3ea5e793cdb1ff39d385c28c47a052fbc426ef4c38: Status 404 returned error can't find the container with id b54c40ffcf6e186b61d03c3ea5e793cdb1ff39d385c28c47a052fbc426ef4c38 Nov 03 09:36:55.432867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:55.432836 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:36:55.432964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:55.432905 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:36:55.432964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:55.432957 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:36:56.136792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:56.136754 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerStarted","Data":"7a4f81a92dec70e37d8774bb78fc309c559ca25568e51d5105fc2bc2e30b654e"} Nov 03 09:36:56.136957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:56.136812 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerStarted","Data":"b54c40ffcf6e186b61d03c3ea5e793cdb1ff39d385c28c47a052fbc426ef4c38"} Nov 03 09:36:56.257780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:56.257728 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podStartSLOduration=2.257712179 podStartE2EDuration="2.257712179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:36:56.255072662 +0000 UTC m=+3439.923113402" watchObservedRunningTime="2026-11-03 09:36:56.257712179 +0000 UTC m=+3439.925752918" Nov 03 09:36:56.955723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:56.955694 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:56.961577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:56.961552 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:57.143207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:57.143173 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:57.144033 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:57.144010 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:36:58.387769 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:58.387738 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:58.993885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:58.993849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:59.176219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:59.176184 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:59.308244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:59.308158 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:36:59.451110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:36:59.451078 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:03.078756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:03.078721 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:03.109692 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:03.109660 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:37:03.300973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:03.300942 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:04.766064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:04.766036 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:05.561410 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:37:05.561363 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:37:06.142463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.142432 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:37:06.147575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.147553 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.213879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.213844 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.213957 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.213982 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214012 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214063 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214053 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214084 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214209 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214236 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.214378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.214334 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.256436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.256404 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:37:06.315026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.314993 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315026 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315079 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315149 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315285 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315329 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315366 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315391 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315442 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315479 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.315571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315503 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.316296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.315786 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.316296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.316069 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.316296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.316172 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.317591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.317564 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.317832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.317813 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.374570 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.374539 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.374933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.374910 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-hxf4z\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.462173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.462089 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:06.889490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.889460 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:37:06.891073 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:37:06.891048 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cfbcbd_bdac_4f3a_94a7_44edd0230829.slice/crio-f38bc6597d4120f9d5fbd632bceafd89fb54f67d80c978d4c81d411d78e72f54 WatchSource:0}: Error finding container f38bc6597d4120f9d5fbd632bceafd89fb54f67d80c978d4c81d411d78e72f54: Status 404 returned error can't find the container with id f38bc6597d4120f9d5fbd632bceafd89fb54f67d80c978d4c81d411d78e72f54 Nov 03 09:37:06.893587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.893555 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:06.893667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.893621 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:06.893667 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:06.893650 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:07.202350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.202252 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" event={"ID":"63cfbcbd-bdac-4f3a-94a7-44edd0230829","Type":"ContainerStarted","Data":"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d"} Nov 03 09:37:07.202350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.202316 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" event={"ID":"63cfbcbd-bdac-4f3a-94a7-44edd0230829","Type":"ContainerStarted","Data":"f38bc6597d4120f9d5fbd632bceafd89fb54f67d80c978d4c81d411d78e72f54"} Nov 03 09:37:07.323238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.323179 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podStartSLOduration=1.323165077 podStartE2EDuration="1.323165077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:07.32185405 +0000 UTC m=+3450.989894788" watchObservedRunningTime="2026-11-03 09:37:07.323165077 +0000 UTC m=+3450.991205815" Nov 03 09:37:07.463171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.463059 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:07.464685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.464661 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:07.464801 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:07.464718 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:08.463444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:08.463411 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:08.463880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:08.463481 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:09.462749 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:09.462715 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:09.462953 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:09.462767 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:09.807831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:09.807738 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:10.463331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:10.463297 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:10.463486 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:10.463357 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:11.463417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:11.463382 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:11.463825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:11.463449 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:12.463428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:12.463395 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:12.463913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:12.463452 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:13.467166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:13.463595 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:13.467166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:13.463656 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:13.532421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:13.532360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:13.668825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:13.668777 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:14.462641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:14.462611 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 03 09:37:14.462825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:14.462671 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 03 09:37:14.816773 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:14.816624 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:15.525810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:15.525779 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:15.526209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:15.526179 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:15.527259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:15.527239 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:37:15.970600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:15.970557 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:16.979128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:16.979049 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:37:16.983079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:16.983063 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.065261 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.065231 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:37:17.074011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.073972 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:17.090212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090191 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872281-openshift-default-dockercfg-4krfp\"" Nov 03 09:37:17.090603 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090582 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.090698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090637 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.090757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090732 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.090816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090796 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.090927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090869 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.090927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090904 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.091014 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.090937 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.091014 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.091008 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.091080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.091030 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwg6w\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192516 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192615 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192681 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192714 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192763 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192845 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.192927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192880 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwg6w\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192947 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.192998 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.193086 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.193131 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.193176 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.193343 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.193687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.193666 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.195629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.195604 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.195780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.195760 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.221624 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.221467 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:17.252406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.252337 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.253698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.253679 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwg6w\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\") pod \"e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.296787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.296754 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:17.731994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.731963 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:37:17.732776 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:37:17.732743 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87449f85_afd4_47a1_ad8d_bee41013ba65.slice/crio-74cbe93d00bc1f3c29761c58934d7d503a63d8548e2332587b436d58018431e5 WatchSource:0}: Error finding container 74cbe93d00bc1f3c29761c58934d7d503a63d8548e2332587b436d58018431e5: Status 404 returned error can't find the container with id 74cbe93d00bc1f3c29761c58934d7d503a63d8548e2332587b436d58018431e5 Nov 03 09:37:17.735409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.735373 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:17.735501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.735467 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:17.735501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:17.735498 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:18.275322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.275287 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerStarted","Data":"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454"} Nov 03 09:37:18.275742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.275337 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerStarted","Data":"74cbe93d00bc1f3c29761c58934d7d503a63d8548e2332587b436d58018431e5"} Nov 03 09:37:18.297418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.297390 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:18.303791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.303766 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:18.397367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.397308 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podStartSLOduration=2.397292938 podStartE2EDuration="2.397292938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:18.395610856 +0000 UTC m=+3462.063651596" watchObservedRunningTime="2026-11-03 09:37:18.397292938 +0000 UTC m=+3462.065333677" Nov 03 09:37:18.590240 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:18.590124 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" podUID="20fdfbfab5aa17ccf7324ae585a08734" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:37:19.280914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:19.280882 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:19.281822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:19.281804 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:37:19.875057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:19.875021 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:20.044989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:20.044955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:21.266117 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.266084 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:37:21.270079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.270059 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.316920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.316892 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:37:21.320776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.320759 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.339354 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339329 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339363 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.339462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339384 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339404 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9cd\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339509 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339538 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339576 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339714 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.339795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339776 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j59m\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.339864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339848 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.339909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339894 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.339952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.339938 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.340023 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340006 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.340057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340048 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.340198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340181 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.340239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340217 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.340319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340306 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.340351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.340333 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.369438 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.369406 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:37:21.421300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.421257 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:37:21.441585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.441551 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.441770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.441608 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j59m\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.441770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.441648 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.441770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.441688 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.441770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.441722 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442051 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442030 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442151 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442100 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442185 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442191 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442235 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442268 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442300 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442335 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442367 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442398 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442430 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442463 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442464 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9cd\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442541 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442597 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442632 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.442746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.442662 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.443171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.443121 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.443171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.443166 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.443345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.443245 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.443427 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.443403 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.443518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.443420 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.445172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.445105 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.445810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.445787 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.445957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.445936 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.446398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.446377 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.501280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.501212 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j59m\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.501853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.501819 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.501960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.501918 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9cd\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7hddr\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.501998 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.501977 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d56dn\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:21.584069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.583990 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:21.634392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:21.634368 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:22.012341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.012315 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:37:22.012881 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:37:22.012848 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4e3da3_f75a_448b_87eb_281f0d724806.slice/crio-3dcb0db1b691999cf114647f3881cc9517d9522acedd996d4a74bb77902247e4 WatchSource:0}: Error finding container 3dcb0db1b691999cf114647f3881cc9517d9522acedd996d4a74bb77902247e4: Status 404 returned error can't find the container with id 3dcb0db1b691999cf114647f3881cc9517d9522acedd996d4a74bb77902247e4 Nov 03 09:37:22.015626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.015598 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.015677 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.015659 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.015708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.015683 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.064202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.064180 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:37:22.068346 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:37:22.068321 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fd6e4e_8a01_4955_b8a8_a0c907791f05.slice/crio-b65f1b24eabfe81112e51960adf2c10e4af7fa4050d88e828c199017666b4371 WatchSource:0}: Error finding container b65f1b24eabfe81112e51960adf2c10e4af7fa4050d88e828c199017666b4371: Status 404 returned error can't find the container with id b65f1b24eabfe81112e51960adf2c10e4af7fa4050d88e828c199017666b4371 Nov 03 09:37:22.071118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.071086 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.071253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.071181 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.071253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.071238 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:37:22.302414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.302314 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" event={"ID":"19fd6e4e-8a01-4955-b8a8-a0c907791f05","Type":"ContainerStarted","Data":"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab"} Nov 03 09:37:22.302414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.302391 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" event={"ID":"19fd6e4e-8a01-4955-b8a8-a0c907791f05","Type":"ContainerStarted","Data":"b65f1b24eabfe81112e51960adf2c10e4af7fa4050d88e828c199017666b4371"} Nov 03 09:37:22.304387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.304358 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" event={"ID":"3c4e3da3-f75a-448b-87eb-281f0d724806","Type":"ContainerStarted","Data":"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392"} Nov 03 09:37:22.304531 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.304407 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" event={"ID":"3c4e3da3-f75a-448b-87eb-281f0d724806","Type":"ContainerStarted","Data":"3dcb0db1b691999cf114647f3881cc9517d9522acedd996d4a74bb77902247e4"} Nov 03 09:37:22.555650 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.555533 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podStartSLOduration=1.5555135359999999 podStartE2EDuration="1.555513536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:22.434407828 +0000 UTC m=+3466.102448564" watchObservedRunningTime="2026-11-03 09:37:22.555513536 +0000 UTC m=+3466.223554274" Nov 03 09:37:22.558255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.558212 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podStartSLOduration=1.558203343 podStartE2EDuration="1.558203343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:22.55321473 +0000 UTC m=+3466.221255466" watchObservedRunningTime="2026-11-03 09:37:22.558203343 +0000 UTC m=+3466.226244081" Nov 03 09:37:22.584452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.584413 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:22.635614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:22.635578 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:23.595924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:23.595886 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:23.596557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:23.596521 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:23.636974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:23.636933 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:23.637170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:23.636994 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:24.586176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:24.585536 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:24.586176 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:24.585598 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:24.635404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:24.635200 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:24.635404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:24.635310 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:24.911167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:24.902267 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:25.590027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:25.589552 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:25.590027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:25.589614 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:25.635076 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:25.635040 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:25.635380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:25.635350 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:26.610166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:26.604264 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:26.610166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:26.604326 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:26.635864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:26.635827 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:26.636038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:26.635875 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:27.597571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:27.597532 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:27.597887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:27.597858 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:27.637177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:27.635604 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:27.637177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:27.635665 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:28.584731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:28.584693 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:28.585039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:28.584761 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:28.635228 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:28.635188 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:28.635442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:28.635260 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:29.215805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.215772 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:37:29.216246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.215988 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" podUID="5e84ee51-3cbd-417a-a2a4-812e522c526a" containerName="maas-api" containerID="cri-o://e0ddc53e98dbf6625180a942baf5480e9c449a34fcc2db557ba9e81309b841e4" gracePeriod=30 Nov 03 09:37:29.310009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.307767 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:29.377133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.377102 4543 generic.go:356] "Generic (PLEG): container finished" podID="5e84ee51-3cbd-417a-a2a4-812e522c526a" containerID="e0ddc53e98dbf6625180a942baf5480e9c449a34fcc2db557ba9e81309b841e4" exitCode=0 Nov 03 09:37:29.377330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.377206 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" event={"ID":"5e84ee51-3cbd-417a-a2a4-812e522c526a","Type":"ContainerDied","Data":"e0ddc53e98dbf6625180a942baf5480e9c449a34fcc2db557ba9e81309b841e4"} Nov 03 09:37:29.515947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.515887 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:37:29.585444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.585399 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 03 09:37:29.585629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.585465 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 03 09:37:29.604826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.604793 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\") pod \"5e84ee51-3cbd-417a-a2a4-812e522c526a\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " Nov 03 09:37:29.604826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.604833 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") pod \"5e84ee51-3cbd-417a-a2a4-812e522c526a\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " Nov 03 09:37:29.605108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.605020 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\" (UniqueName: \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\") pod \"5e84ee51-3cbd-417a-a2a4-812e522c526a\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " Nov 03 09:37:29.605405 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.605376 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\") pod \"5e84ee51-3cbd-417a-a2a4-812e522c526a\" (UID: \"5e84ee51-3cbd-417a-a2a4-812e522c526a\") " Nov 03 09:37:29.605925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.605898 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca" pod "5e84ee51-3cbd-417a-a2a4-812e522c526a" (UID: "5e84ee51-3cbd-417a-a2a4-812e522c526a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:29.607492 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.607470 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e84ee51-3cbd-417a-a2a4-812e522c526a-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:29.607876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.607855 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b" pod "5e84ee51-3cbd-417a-a2a4-812e522c526a" (UID: "5e84ee51-3cbd-417a-a2a4-812e522c526a"). InnerVolumeSpecName "kube-api-access-tk58b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:29.608121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.608102 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls" pod "5e84ee51-3cbd-417a-a2a4-812e522c526a" (UID: "5e84ee51-3cbd-417a-a2a4-812e522c526a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:37:29.608216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.608197 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs" pod "5e84ee51-3cbd-417a-a2a4-812e522c526a" (UID: "5e84ee51-3cbd-417a-a2a4-812e522c526a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:37:29.651203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.650768 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 03 09:37:29.651203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.650822 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 03 09:37:29.708637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.708597 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk58b\" (UniqueName: \"kubernetes.io/projected/5e84ee51-3cbd-417a-a2a4-812e522c526a-kube-api-access-tk58b\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:29.708637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.708634 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:29.708844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:29.708649 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e84ee51-3cbd-417a-a2a4-812e522c526a-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:30.389912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.389878 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" event={"ID":"5e84ee51-3cbd-417a-a2a4-812e522c526a","Type":"ContainerDied","Data":"645f1ede1ebafb67b86f9de2094e15c1670bfb357a7dc5a8600cdcb6a288e5a5"} Nov 03 09:37:30.390581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.390558 4543 scope.go:122] "RemoveContainer" containerID="e0ddc53e98dbf6625180a942baf5480e9c449a34fcc2db557ba9e81309b841e4" Nov 03 09:37:30.390812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.390390 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn" Nov 03 09:37:30.564247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.564210 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:37:30.614917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.614874 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03a6e5-7b6b7d6c87-sklrn"] Nov 03 09:37:30.790275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.790196 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:30.790672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.790651 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:30.791765 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.791744 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:37:30.825379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.825348 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:30.827355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.825931 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:30.827355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:30.826695 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:37:31.131093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:31.131062 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:32.451385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:32.451353 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e84ee51-3cbd-417a-a2a4-812e522c526a" path="/var/lib/kubelet/pods/5e84ee51-3cbd-417a-a2a4-812e522c526a/volumes" Nov 03 09:37:33.241846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:33.241815 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:33.453763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:33.453722 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:33.656597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:33.656557 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:33.944435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:33.944401 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:34.094074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:34.094044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:35.533165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:35.532570 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:35.701662 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:37:35.701585 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:37:36.447165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:36.412617 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:36.856223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:36.842214 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:37:36.883932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:36.868028 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" containerID="cri-o://564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554" gracePeriod=30 Nov 03 09:37:36.883932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:36.868451 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:37.109929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:37.109790 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:37.197162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:37.190774 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:37:38.118221 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:38.118190 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:38.325447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:38.325385 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:39.073734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:39.073695 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:39.121065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:39.121030 4543 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:37:39.121453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:39.121095 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:37:41.953095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:41.953062 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:42.113373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.113351 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:37:42.160355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160320 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160452 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160508 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160574 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160611 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160695 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160790 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160828 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160877 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\") pod \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\" (UID: \"ebdaccee-0f81-4e30-b7c3-f3511386cd78\") " Nov 03 09:37:42.160987 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.160969 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:37:42.161240 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161220 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:37:42.161572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161547 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:37:42.161661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161638 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:37:42.161661 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161651 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:37:42.161867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161810 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.161867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161829 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.161867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161842 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.161867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161854 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.161867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.161866 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.163151 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.163111 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "kube-api-access-gn9kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:42.163419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.163399 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:37:42.163553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.163534 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:37:42.163797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.163778 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token" pod "ebdaccee-0f81-4e30-b7c3-f3511386cd78" (UID: "ebdaccee-0f81-4e30-b7c3-f3511386cd78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:37:42.263718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.263641 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.263718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.263676 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.263718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.263687 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9kr\" (UniqueName: \"kubernetes.io/projected/ebdaccee-0f81-4e30-b7c3-f3511386cd78-kube-api-access-gn9kr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.263718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.263696 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebdaccee-0f81-4e30-b7c3-f3511386cd78-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:37:42.291054 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.291007 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:42.423205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.423169 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:42.778485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.778454 4543 generic.go:356] "Generic (PLEG): container finished" podID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerID="564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554" exitCode=0 Nov 03 09:37:42.778691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.778504 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" Nov 03 09:37:42.778691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.778535 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerDied","Data":"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554"} Nov 03 09:37:42.778691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.778603 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh" event={"ID":"ebdaccee-0f81-4e30-b7c3-f3511386cd78","Type":"ContainerDied","Data":"1803f052a946e3a63920ae544276f1f41e18de10d785ef8da3320250eb148518"} Nov 03 09:37:42.778691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.778634 4543 scope.go:122] "RemoveContainer" containerID="564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554" Nov 03 09:37:42.788901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.788870 4543 scope.go:122] "RemoveContainer" containerID="5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5" Nov 03 09:37:42.797903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.797885 4543 scope.go:122] "RemoveContainer" containerID="564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554" Nov 03 09:37:42.798220 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:37:42.798198 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554\": container with ID starting with 564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554 not found: ID does not exist" containerID="564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554" Nov 03 09:37:42.798317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.798226 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554"} err="failed to get container status \"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554\": rpc error: code = NotFound desc = could not find container \"564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554\": container with ID starting with 564d9ff59e0387bb1c92903eb5039e12fd27027196ca21007261e8cb90db4554 not found: ID does not exist" Nov 03 09:37:42.798317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.798245 4543 scope.go:122] "RemoveContainer" containerID="5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5" Nov 03 09:37:42.798520 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:37:42.798502 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5\": container with ID starting with 5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5 not found: ID does not exist" containerID="5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5" Nov 03 09:37:42.798581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.798527 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5"} err="failed to get container status \"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5\": rpc error: code = NotFound desc = could not find container \"5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5\": container with ID starting with 5558127fa6388530e929f58cef79eb9c4fc6c731ac88fc8522ec4e90e6944cc5 not found: ID does not exist" Nov 03 09:37:42.976109 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:42.976068 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:37:43.010374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:43.010330 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03a6e5-openshift-default-55b765c7ff-f25wh"] Nov 03 09:37:43.872037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:43.872003 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:44.016184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:44.010993 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:44.451644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:44.451610 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" path="/var/lib/kubelet/pods/ebdaccee-0f81-4e30-b7c3-f3511386cd78/volumes" Nov 03 09:37:49.133361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:49.133328 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:50.143062 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:50.143027 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:50.277114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:50.277084 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:51.738360 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:51.738326 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:52.253863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:52.253828 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:54.352576 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:54.352545 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:59.371442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:59.371404 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:37:59.520349 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:37:59.520314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:03.061820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:03.061784 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:04.329082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:04.329034 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:05.560733 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:05.560699 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:38:09.333122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:09.333092 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:13.992495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.992462 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:38:13.993083 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993068 4543 cpu_manager.go:414] "containerMap: removing container" podUID="5e84ee51-3cbd-417a-a2a4-812e522c526a" containerName="maas-api" Nov 03 09:38:13.993127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993084 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84ee51-3cbd-417a-a2a4-812e522c526a" containerName="maas-api" Nov 03 09:38:13.993127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993111 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.993127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993117 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.993248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993131 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.993248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993149 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.993248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993240 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84ee51-3cbd-417a-a2a4-812e522c526a" containerName="maas-api" Nov 03 09:38:13.993340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993253 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.993340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.993261 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdaccee-0f81-4e30-b7c3-f3511386cd78" containerName="istio-proxy" Nov 03 09:38:13.996596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:13.996581 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.047418 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.047386 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtshw\" (UniqueName: \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.047592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.047531 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.047662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.047636 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.047799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.047774 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.100875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.100841 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-373116-serving-cert\"" Nov 03 09:38:14.103821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.103800 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:38:14.149543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.149511 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.149725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.149598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtshw\" (UniqueName: \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.149725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.149687 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.149852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.149771 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.149852 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:14.149788 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-serving-cert: secret "maas-api-e2e-extmod-a-373116-serving-cert" not found Nov 03 09:38:14.149968 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:14.149946 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls podName:61e8d938-bc22-4ee1-b4aa-a4b7fe603b31 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:14.649926717 +0000 UTC m=+3518.317967450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls") pod "maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" (UID: "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31") : secret "maas-api-e2e-extmod-a-373116-serving-cert" not found Nov 03 09:38:14.150340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.150301 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.153041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.153004 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.177232 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.177203 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:14.210664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.210642 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtshw\" (UniqueName: \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.345627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.345551 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:14.470874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.470841 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:14.657304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.657272 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.659872 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.659843 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") pod \"maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:14.909989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:14.909915 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:15.285012 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:38:15.284981 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e8d938_bc22_4ee1_b4aa_a4b7fe603b31.slice/crio-0dc630871752cdaa07785addea135a259d7d05b4530130e7ee7d7111129ad84e WatchSource:0}: Error finding container 0dc630871752cdaa07785addea135a259d7d05b4530130e7ee7d7111129ad84e: Status 404 returned error can't find the container with id 0dc630871752cdaa07785addea135a259d7d05b4530130e7ee7d7111129ad84e Nov 03 09:38:15.285459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:15.285436 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:38:15.286636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:15.286618 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:38:15.799846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:15.799808 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:15.953509 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:15.953469 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:15.983082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:15.983047 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" event={"ID":"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31","Type":"ContainerStarted","Data":"0dc630871752cdaa07785addea135a259d7d05b4530130e7ee7d7111129ad84e"} Nov 03 09:38:16.995069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:16.995033 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" event={"ID":"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31","Type":"ContainerStarted","Data":"0397f0131a3c16fb816bcb9558dea4f517192abf9c30ba6b1642c9593f0d99e3"} Nov 03 09:38:16.995585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:16.995168 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:17.116776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:17.116720 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" podStartSLOduration=3.256262179 podStartE2EDuration="4.116692036s" totalImagesPullingTime="860.429857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:38:17.11385373 +0000 UTC m=+3520.781894464" watchObservedRunningTime="2026-11-03 09:38:17.116692036 +0000 UTC m=+3520.784732790" Nov 03 09:38:19.620343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:19.620308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:19.899991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:19.899890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:20.082794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:20.082762 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:20.223436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:20.223349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:20.398321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:20.398286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:23.006832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:23.006802 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:38:23.197827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:23.197767 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:23.391426 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:23.391399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:24.591340 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:24.591311 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:24.591340 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:24.591335 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:24.591864 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:24.591426 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:25.091410236 +0000 UTC m=+3528.759450952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:25.045320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:25.045290 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:25.097199 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:25.097173 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:25.097199 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:25.097193 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:25.097412 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:25.097273 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:26.097255262 +0000 UTC m=+3529.765295979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:26.109687 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:26.109655 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:26.109687 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:26.109680 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:26.110099 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:26.109733 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:28.109718462 +0000 UTC m=+3531.777759178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:28.130954 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:28.130923 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:28.130954 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:28.130949 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:28.131420 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:28.131013 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:32.13099836 +0000 UTC m=+3535.799039076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:30.086876 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:30.086836 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:32.094726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:32.094687 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:32.180933 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:32.180905 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:32.180933 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:32.180926 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:32.181185 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:32.180976 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:40.180962052 +0000 UTC m=+3543.849002768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:35.032205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:35.032174 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:35.561382 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:35.561333 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:38:40.032553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:40.032514 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:40.208700 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:40.208661 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:40.208700 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:40.208686 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:40.208922 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:40.208746 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:38:56.208731558 +0000 UTC m=+3559.876772274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:43.186267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:43.186234 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:38:43.186707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:43.186307 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:38:44.170317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.170237 4543 generic.go:356] "Generic (PLEG): container finished" podID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerID="7a4f81a92dec70e37d8774bb78fc309c559ca25568e51d5105fc2bc2e30b654e" exitCode=0 Nov 03 09:38:44.170527 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.170306 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerDied","Data":"7a4f81a92dec70e37d8774bb78fc309c559ca25568e51d5105fc2bc2e30b654e"} Nov 03 09:38:44.170717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.170704 4543 scope.go:122] "RemoveContainer" containerID="7a4f81a92dec70e37d8774bb78fc309c559ca25568e51d5105fc2bc2e30b654e" Nov 03 09:38:44.172193 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.172168 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:38:44.172262 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.172235 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:38:44.172300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.172268 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:38:44.977616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:44.977577 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:45.178674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:45.178645 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerStarted","Data":"f8dc07b1fe06d60a0ade71464becf2353e8176f060908128adc0ca218f6f0522"} Nov 03 09:38:45.955757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:45.955720 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:38:45.955948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:45.955902 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 09:38:45.956003 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:45.955950 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 09:38:46.955647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:46.955566 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 09:38:46.955647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:46.955616 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 09:38:47.955662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:47.955626 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 09:38:47.956037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:47.955675 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 09:38:48.955496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:48.955459 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 09:38:48.955670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:48.955518 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 09:38:49.934583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:49.934544 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:49.955810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:49.955779 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 09:38:49.955951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:49.955836 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 09:38:50.960114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:50.960080 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:38:50.960614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:50.960591 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:38:50.961373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:50.961353 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:38:54.900455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:54.900418 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:38:55.009376 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.009342 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:55.358338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.358240 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:38:55.358517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.358368 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.367526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.367491 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:55.415571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.415539 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-872281-serving-cert\"" Nov 03 09:38:55.456007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.455949 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.456243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.456074 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6kk\" (UniqueName: \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.456243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.456219 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.456395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.456338 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.558130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.558087 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6kk\" (UniqueName: \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.558355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.558191 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.558780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.558759 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.559374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.559352 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.560282 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.560259 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.561881 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.561827 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.562549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.562526 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.619026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.618994 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6kk\" (UniqueName: \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\") pod \"maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:55.648664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.648628 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:38:55.693870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:55.693838 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:56.108524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:56.108500 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:38:56.109266 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:38:56.109244 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a44fe82_9028_4606_9c20_70f308e048b7.slice/crio-ea49d64e1c798d02424d62f51b0bdd77502c84499057bf0115bb664fca0dd2af WatchSource:0}: Error finding container ea49d64e1c798d02424d62f51b0bdd77502c84499057bf0115bb664fca0dd2af: Status 404 returned error can't find the container with id ea49d64e1c798d02424d62f51b0bdd77502c84499057bf0115bb664fca0dd2af Nov 03 09:38:56.270251 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:56.270223 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:56.270251 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:56.270245 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:56.270456 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:38:56.270298 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:39:28.270284191 +0000 UTC m=+3591.938324907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:38:56.375874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:56.375837 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" event={"ID":"8a44fe82-9028-4606-9c20-70f308e048b7","Type":"ContainerStarted","Data":"ea49d64e1c798d02424d62f51b0bdd77502c84499057bf0115bb664fca0dd2af"} Nov 03 09:38:57.383958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:57.383918 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" event={"ID":"8a44fe82-9028-4606-9c20-70f308e048b7","Type":"ContainerStarted","Data":"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85"} Nov 03 09:38:57.384389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:57.384049 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:38:57.500934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:57.500884 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" podStartSLOduration=2.641121949 podStartE2EDuration="3.50086959s" totalImagesPullingTime="859.747641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:38:57.499843155 +0000 UTC m=+3561.167883894" watchObservedRunningTime="2026-11-03 09:38:57.50086959 +0000 UTC m=+3561.168910328" Nov 03 09:38:59.867287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:38:59.867253 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:00.228259 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:00.228156 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:03.396278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:03.396246 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:39:03.745958 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:03.745880 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:05.247156 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:05.247108 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:05.561477 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:05.561394 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:39:10.271444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:10.271411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:15.234386 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:15.234350 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:17.847991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:17.847958 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:18.006760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:18.006727 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:19.897523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:19.897485 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:20.335123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.335036 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:20.698629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.698577 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:39:20.706414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.706387 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.734634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.734598 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:20.757701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.757668 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:39:20.804757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.804725 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8204d6-openshift-default-dockercfg-g725c\"" Nov 03 09:39:20.822227 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.822194 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvzkp\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.822482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.822462 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.822693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.822673 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.822823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.822805 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.822961 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.822943 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.823286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.823265 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.823468 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.823448 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.823591 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.823575 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.823757 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.823740 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.879318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.879285 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:20.925189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.925157 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.925683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.925666 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.925767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.925757 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.925879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.925870 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926022 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926013 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvzkp\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926089 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926199 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926214 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926307 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926400 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926506 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926495 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926560 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926536 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.926774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.926754 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.928651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.928621 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:20.928917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:20.928901 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:21.006355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.004183 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvzkp\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:21.009629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.009590 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\") pod \"e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:21.022863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.022837 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:21.253915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.253878 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:21.473885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.473859 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:39:21.475276 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:39:21.475251 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c422f0_cc6a_4023_9983_ca1fd2f03248.slice/crio-8275403b1916b1b0135c4adf6ccd407a4e97d237bdae5b9b31b1ea994fc953e4 WatchSource:0}: Error finding container 8275403b1916b1b0135c4adf6ccd407a4e97d237bdae5b9b31b1ea994fc953e4: Status 404 returned error can't find the container with id 8275403b1916b1b0135c4adf6ccd407a4e97d237bdae5b9b31b1ea994fc953e4 Nov 03 09:39:21.477853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.477820 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:21.477936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.477893 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:21.477936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.477925 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:21.543706 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:21.543675 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerStarted","Data":"8275403b1916b1b0135c4adf6ccd407a4e97d237bdae5b9b31b1ea994fc953e4"} Nov 03 09:39:22.560526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:22.560487 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerStarted","Data":"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045"} Nov 03 09:39:22.734198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:22.734108 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podStartSLOduration=2.734091506 podStartE2EDuration="2.734091506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:22.731185414 +0000 UTC m=+3586.399226152" watchObservedRunningTime="2026-11-03 09:39:22.734091506 +0000 UTC m=+3586.402132244" Nov 03 09:39:23.023028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:23.022992 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:23.028805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:23.028777 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:23.567044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:23.567011 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:23.568189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:23.568155 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:39:25.583316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:25.583286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:26.177935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.177885 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:26.391547 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.391503 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:26.551869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.551790 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:26.589255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.589221 4543 generic.go:356] "Generic (PLEG): container finished" podID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerID="581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454" exitCode=0 Nov 03 09:39:26.589694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.589295 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerDied","Data":"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454"} Nov 03 09:39:26.589767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.589720 4543 scope.go:122] "RemoveContainer" containerID="581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454" Nov 03 09:39:26.591352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.591326 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:26.591453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.591390 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:26.591453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:26.591422 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:39:27.610795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:27.610762 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerStarted","Data":"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1"} Nov 03 09:39:28.300165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:28.299513 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:39:28.300165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:28.299638 4543 patch_prober.go:29] interesting pod/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 09:39:28.300165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:28.299679 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 09:39:28.328204 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:28.328173 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-546e9f5ec78c: secret "agc-cred-546e9f5ec78c" not found Nov 03 09:39:28.328204 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:28.328205 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:39:28.328402 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:28.328220 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:39:28.328402 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:28.328279 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:40:32.328264988 +0000 UTC m=+3655.996305719 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:39:29.302166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:29.301508 4543 patch_prober.go:29] interesting pod/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 09:39:29.302166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:29.301584 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 09:39:30.304171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:30.301616 4543 patch_prober.go:29] interesting pod/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 09:39:30.304171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:30.301675 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 09:39:30.747273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:30.747240 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:30.915166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:30.912782 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:31.051964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:31.051871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:31.309223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:31.308685 4543 patch_prober.go:29] interesting pod/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 09:39:31.309223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:31.308745 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 09:39:32.306167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:32.299495 4543 patch_prober.go:29] interesting pod/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Nov 03 09:39:32.306167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:32.299565 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Nov 03 09:39:33.301789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:33.301718 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:39:33.302160 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:33.302039 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:39:33.302701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:33.302683 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:39:35.561353 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:39:35.561316 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:39:36.367820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:36.367783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:36.958529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:36.958497 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:39:37.019315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:37.019285 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:39:41.703391 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:41.703357 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:41.867885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:41.867854 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:41.996293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:41.996215 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:47.447369 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:47.447335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:49.148869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:49.148834 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:52.421840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:52.421789 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:39:57.396873 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:39:57.396816 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:00.216910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:00.216847 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:00.430633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:00.430596 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:01.161976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:01.161938 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:01.535618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:01.535529 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:02.451257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:02.451222 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:04.093158 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:04.093104 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:05.561370 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:05.561324 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:40:05.760797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:05.760763 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:05.942213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:05.942183 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:06.090125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:06.090096 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:07.881967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:07.881930 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:09.585286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:09.585251 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:09.719669 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:09.719632 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:12.951523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:12.951488 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:17.861568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:17.861490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:21.100819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:21.100781 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:22.811716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:22.811674 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:27.855365 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:27.855335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:28.679435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:28.679403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:29.095583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.095502 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:29.630582 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.630544 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:40:29.634914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.634893 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.674921 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675058 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675163 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675096 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfm6\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675242 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675272 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675296 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675337 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.675571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.675375 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.731676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.731646 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-af514d-openshift-default-dockercfg-ldlbz\"" Nov 03 09:40:29.745043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.744272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:29.765748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.765719 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:40:29.777205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777129 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.777459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777437 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.777633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777615 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfm6\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777767 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777807 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777836 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777884 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777911 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.777926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.778060 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.778283 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.778459 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.778874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.778673 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.779106 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.779074 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.781732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.781690 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.788596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.788569 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.838483 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.838457 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.838681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.838658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfm6\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\") pod \"e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:29.949622 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:29.949535 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:30.113940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:30.113901 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:30.384685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:30.384654 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:40:30.385853 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:40:30.385809 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df46ae1_d457_4d77_a6ae_c5c7949309e4.slice/crio-77eb03a2d2c68b6971fe7501ca838b491b6fab3a45e340b7451f4bfda7f6711d WatchSource:0}: Error finding container 77eb03a2d2c68b6971fe7501ca838b491b6fab3a45e340b7451f4bfda7f6711d: Status 404 returned error can't find the container with id 77eb03a2d2c68b6971fe7501ca838b491b6fab3a45e340b7451f4bfda7f6711d Nov 03 09:40:30.388350 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:30.388321 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:40:30.388420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:30.388385 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:40:30.388460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:30.388417 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:40:31.052685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:31.052645 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerStarted","Data":"892d149c49ab144056187de6d49edcfb97dc3b3878e95934da1cb80cbc5ae961"} Nov 03 09:40:31.052852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:31.052705 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerStarted","Data":"77eb03a2d2c68b6971fe7501ca838b491b6fab3a45e340b7451f4bfda7f6711d"} Nov 03 09:40:31.224434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:31.224384 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podStartSLOduration=2.224370563 podStartE2EDuration="2.224370563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:31.222079434 +0000 UTC m=+3654.890120563" watchObservedRunningTime="2026-11-03 09:40:31.224370563 +0000 UTC m=+3654.892411300" Nov 03 09:40:31.949653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:31.949621 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:31.956031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:31.956005 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:32.059406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:32.059379 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:32.060335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:32.060318 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:40:32.414894 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:32.414865 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-546e9f5ec78c: secret "agc-cred-546e9f5ec78c" not found Nov 03 09:40:32.414894 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:32.414888 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:40:32.414894 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:32.414897 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:40:32.415386 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:32.414946 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:42:34.414933738 +0000 UTC m=+3778.082974455 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:40:33.518385 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:33.518354 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:33.948981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:33.948949 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:34.138474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:34.138442 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:34.270884 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:34.270773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:35.562356 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:35.562323 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:40:37.257170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:37.257122 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:37.437915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:37.436849 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:39.072206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:39.072166 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:44.274315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:44.274282 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:49.244843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:49.244807 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:52.646025 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:52.645990 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:52.780722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:52.780688 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:53.995045 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:53.995013 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:40:53.999027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:53.999009 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.103471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.103443 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8204d6-serving-cert\"" Nov 03 09:40:54.106505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.106480 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.106628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.106606 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.106770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.106750 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.106817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.106786 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlhg\" (UniqueName: \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.106870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.106856 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:40:54.181977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.181947 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:54.208585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.208554 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.208585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.208585 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlhg\" (UniqueName: \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.208859 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:54.208713 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-serving-cert: secret "maas-api-e2e-inf-a-8204d6-serving-cert" not found Nov 03 09:40:54.208859 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:40:54.208783 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls podName:967c1242-1899-4a31-9d42-3ce0fe87dc79 nodeName:}" failed. No retries permitted until 2026-11-03 09:40:54.708763755 +0000 UTC m=+3678.376804482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls") pod "maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" (UID: "967c1242-1899-4a31-9d42-3ce0fe87dc79") : secret "maas-api-e2e-inf-a-8204d6-serving-cert" not found Nov 03 09:40:54.208859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.208804 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.209019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.208902 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.209462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.209437 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.211501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.211478 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.273346 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.273265 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlhg\" (UniqueName: \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.356431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.356392 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:54.535981 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.535885 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:40:54.715273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.715242 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.717718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.717700 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") pod \"maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:54.912742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:54.912708 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:55.290910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:55.290879 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:40:55.291365 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:40:55.291342 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967c1242_1899_4a31_9d42_3ce0fe87dc79.slice/crio-6d06744f7aee6f878778d554aea37ce68c8678c6f0911177b89fc38c6cfb87bb WatchSource:0}: Error finding container 6d06744f7aee6f878778d554aea37ce68c8678c6f0911177b89fc38c6cfb87bb: Status 404 returned error can't find the container with id 6d06744f7aee6f878778d554aea37ce68c8678c6f0911177b89fc38c6cfb87bb Nov 03 09:40:56.214822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:56.214791 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" event={"ID":"967c1242-1899-4a31-9d42-3ce0fe87dc79","Type":"ContainerStarted","Data":"6d06744f7aee6f878778d554aea37ce68c8678c6f0911177b89fc38c6cfb87bb"} Nov 03 09:40:57.225511 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:57.225475 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" event={"ID":"967c1242-1899-4a31-9d42-3ce0fe87dc79","Type":"ContainerStarted","Data":"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f"} Nov 03 09:40:57.225991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:57.225645 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:40:57.344631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:57.344581 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" podStartSLOduration=3.444477777 podStartE2EDuration="4.344567272s" totalImagesPullingTime="900.089495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:57.342841286 +0000 UTC m=+3681.010882024" watchObservedRunningTime="2026-11-03 09:40:57.344567272 +0000 UTC m=+3681.012608010" Nov 03 09:40:59.353074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:40:59.353034 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:00.560314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:00.560282 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:03.241194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:03.241161 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:41:04.309305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:04.309273 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:04.440819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:04.440792 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:05.561853 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:41:05.561811 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:41:09.304078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:09.304045 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:14.267911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:14.267871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:19.352655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:19.352615 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:21.433745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:21.433707 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:23.638097 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:23.638059 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:24.728691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:24.728654 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:29.457894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.457863 4543 generic.go:356] "Generic (PLEG): container finished" podID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerID="7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045" exitCode=0 Nov 03 09:41:29.458351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.457909 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerDied","Data":"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045"} Nov 03 09:41:29.458351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.458271 4543 scope.go:122] "RemoveContainer" containerID="7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045" Nov 03 09:41:29.459804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.459773 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:41:29.459912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.459841 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:41:29.459912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:29.459871 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:41:30.066469 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:30.066434 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:30.469209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:30.469122 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerStarted","Data":"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497"} Nov 03 09:41:30.507601 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:30.507566 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:30.643466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:30.643424 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:31.023998 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:31.023964 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:41:31.024226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:31.024187 4543 patch_prober.go:29] interesting pod/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 09:41:31.024311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:31.024231 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 09:41:32.023310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:32.023271 4543 patch_prober.go:29] interesting pod/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 09:41:32.023803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:32.023325 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 09:41:33.023375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:33.023350 4543 patch_prober.go:29] interesting pod/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 09:41:33.023651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:33.023409 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 09:41:34.023783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:34.023742 4543 patch_prober.go:29] interesting pod/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 09:41:34.024271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:34.023801 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 09:41:35.032108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:35.028223 4543 patch_prober.go:29] interesting pod/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 03 09:41:35.032108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:35.028296 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 03 09:41:35.562640 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:41:35.562600 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:41:35.599356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:35.599324 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:36.028700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:36.028669 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:41:36.029180 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:36.029129 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:41:36.029792 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:36.029775 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:41:36.989272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:36.989231 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:40.840289 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:40.840256 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:41.000797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:41.000761 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:41.128737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:41.128698 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:46.333196 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:46.333160 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:51.325553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:51.325423 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:41:56.286637 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:41:56.286579 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:00.902554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:00.902519 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:01.236925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:01.236809 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:04.779064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:04.778998 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:05.561337 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:05.561302 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:42:06.058672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:06.058634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:06.186466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:06.186424 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:06.316894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:06.316817 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:11.183769 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:11.183729 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:11.348351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:11.348310 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:11.475746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:11.475662 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:16.303791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:16.303760 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:21.263053 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:21.263010 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:21.784612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:21.784579 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:26.248903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:26.248873 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:29.106756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:29.106711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:31.241289 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:31.241257 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:34.488580 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:34.488548 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-546e9f5ec78c: secret "agc-cred-546e9f5ec78c" not found Nov 03 09:42:34.488580 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:34.488574 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:42:34.488580 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:34.488582 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:42:34.489048 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:34.488644 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:44:36.488623559 +0000 UTC m=+3900.156664275 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : [secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:42:34.979733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:34.979697 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:35.113444 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.113408 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:35.514805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.514760 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:42:35.515223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.515032 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" podUID="967c1242-1899-4a31-9d42-3ce0fe87dc79" containerName="maas-api" containerID="cri-o://0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f" gracePeriod=30 Nov 03 09:42:35.577437 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:35.577386 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:42:35.611628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.611599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:35.793040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.793014 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:42:35.814060 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.814033 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:35.890087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.890055 4543 generic.go:356] "Generic (PLEG): container finished" podID="967c1242-1899-4a31-9d42-3ce0fe87dc79" containerID="0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f" exitCode=0 Nov 03 09:42:35.890302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.890107 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" Nov 03 09:42:35.890302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.890134 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" event={"ID":"967c1242-1899-4a31-9d42-3ce0fe87dc79","Type":"ContainerDied","Data":"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f"} Nov 03 09:42:35.890302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.890211 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk" event={"ID":"967c1242-1899-4a31-9d42-3ce0fe87dc79","Type":"ContainerDied","Data":"6d06744f7aee6f878778d554aea37ce68c8678c6f0911177b89fc38c6cfb87bb"} Nov 03 09:42:35.890302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.890234 4543 scope.go:122] "RemoveContainer" containerID="0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f" Nov 03 09:42:35.901540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.901498 4543 scope.go:122] "RemoveContainer" containerID="0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f" Nov 03 09:42:35.901957 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:35.901930 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f\": container with ID starting with 0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f not found: ID does not exist" containerID="0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f" Nov 03 09:42:35.902071 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.901969 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f"} err="failed to get container status \"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f\": rpc error: code = NotFound desc = could not find container \"0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f\": container with ID starting with 0334c8adceb22e10034ff9a723da0efc138989bf8ec1ca161250d6ea7bfc499f not found: ID does not exist" Nov 03 09:42:35.909371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.909340 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") pod \"967c1242-1899-4a31-9d42-3ce0fe87dc79\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " Nov 03 09:42:35.909501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.909460 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\" (UniqueName: \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\") pod \"967c1242-1899-4a31-9d42-3ce0fe87dc79\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " Nov 03 09:42:35.909658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.909639 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\") pod \"967c1242-1899-4a31-9d42-3ce0fe87dc79\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " Nov 03 09:42:35.909736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.909676 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\") pod \"967c1242-1899-4a31-9d42-3ce0fe87dc79\" (UID: \"967c1242-1899-4a31-9d42-3ce0fe87dc79\") " Nov 03 09:42:35.911105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.911073 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca" pod "967c1242-1899-4a31-9d42-3ce0fe87dc79" (UID: "967c1242-1899-4a31-9d42-3ce0fe87dc79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:42:35.973452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.973415 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs" pod "967c1242-1899-4a31-9d42-3ce0fe87dc79" (UID: "967c1242-1899-4a31-9d42-3ce0fe87dc79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:42:35.975495 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.975463 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls" pod "967c1242-1899-4a31-9d42-3ce0fe87dc79" (UID: "967c1242-1899-4a31-9d42-3ce0fe87dc79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:42:35.982404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:35.982364 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg" pod "967c1242-1899-4a31-9d42-3ce0fe87dc79" (UID: "967c1242-1899-4a31-9d42-3ce0fe87dc79"). InnerVolumeSpecName "kube-api-access-2hlhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:42:36.011022 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.010994 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/967c1242-1899-4a31-9d42-3ce0fe87dc79-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:36.011022 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.011019 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:36.011249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.011030 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/967c1242-1899-4a31-9d42-3ce0fe87dc79-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:36.011249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.011038 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hlhg\" (UniqueName: \"kubernetes.io/projected/967c1242-1899-4a31-9d42-3ce0fe87dc79-kube-api-access-2hlhg\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:36.322633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.322593 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:42:36.393780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.393468 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8204d6-cb5c8fbff-hswdk"] Nov 03 09:42:36.467602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:36.467510 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967c1242-1899-4a31-9d42-3ce0fe87dc79" path="/var/lib/kubelet/pods/967c1242-1899-4a31-9d42-3ce0fe87dc79/volumes" Nov 03 09:42:37.495707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:37.495676 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:41.922017 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:41.921984 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:42.055321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:42.055286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:42.478729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:42.478697 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:43.522989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:43.522957 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:44.124185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:44.124112 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:44.152603 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:44.152574 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:42:44.152821 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:44.152777 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" containerID="cri-o://f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497" gracePeriod=30 Nov 03 09:42:47.975204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:47.975168 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:48.100325 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.100283 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 09:42:48.102320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.100362 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 09:42:48.984009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.983978 4543 generic.go:356] "Generic (PLEG): container finished" podID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerID="892d149c49ab144056187de6d49edcfb97dc3b3878e95934da1cb80cbc5ae961" exitCode=0 Nov 03 09:42:48.984395 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.984052 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerDied","Data":"892d149c49ab144056187de6d49edcfb97dc3b3878e95934da1cb80cbc5ae961"} Nov 03 09:42:48.984480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.984467 4543 scope.go:122] "RemoveContainer" containerID="892d149c49ab144056187de6d49edcfb97dc3b3878e95934da1cb80cbc5ae961" Nov 03 09:42:48.986059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.986034 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:42:48.986132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.986113 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:42:48.986189 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:48.986164 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:42:49.435945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.435911 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:42:49.516399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516369 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516466 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516598 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516653 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516671 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:42:49.516755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516748 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516794 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.516950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516842 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:42:49.516950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516913 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:42:49.517101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.516936 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\") pod \"93c422f0-cc6a-4023-9983-ca1fd2f03248\" (UID: \"93c422f0-cc6a-4023-9983-ca1fd2f03248\") " Nov 03 09:42:49.517101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517001 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:42:49.517101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517045 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:42:49.517618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517592 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.517688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517627 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93c422f0-cc6a-4023-9983-ca1fd2f03248-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.517804 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517734 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.517866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517808 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.517866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.517823 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.518859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.518832 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:42:49.519001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.518887 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "kube-api-access-mvzkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:42:49.519204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.519185 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:42:49.519311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.519275 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy" pod "93c422f0-cc6a-4023-9983-ca1fd2f03248" (UID: "93c422f0-cc6a-4023-9983-ca1fd2f03248"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:42:49.619090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.619057 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.619090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.619084 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.619090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.619093 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvzkp\" (UniqueName: \"kubernetes.io/projected/93c422f0-cc6a-4023-9983-ca1fd2f03248-kube-api-access-mvzkp\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.619338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.619102 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93c422f0-cc6a-4023-9983-ca1fd2f03248-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:42:49.992657 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.992612 4543 generic.go:356] "Generic (PLEG): container finished" podID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerID="f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497" exitCode=0 Nov 03 09:42:49.993197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.992655 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerDied","Data":"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497"} Nov 03 09:42:49.993197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.992716 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" event={"ID":"93c422f0-cc6a-4023-9983-ca1fd2f03248","Type":"ContainerDied","Data":"8275403b1916b1b0135c4adf6ccd407a4e97d237bdae5b9b31b1ea994fc953e4"} Nov 03 09:42:49.993197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.992667 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w" Nov 03 09:42:49.993197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.992737 4543 scope.go:122] "RemoveContainer" containerID="f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497" Nov 03 09:42:49.995169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:49.995112 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerStarted","Data":"e7510b8e500653432090b56809bfaaf78fde6a7783c9e94701d7e53103bdb8f4"} Nov 03 09:42:50.009318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.009290 4543 scope.go:122] "RemoveContainer" containerID="7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045" Nov 03 09:42:50.019335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.019317 4543 scope.go:122] "RemoveContainer" containerID="f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497" Nov 03 09:42:50.020554 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:50.020524 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497\": container with ID starting with f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497 not found: ID does not exist" containerID="f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497" Nov 03 09:42:50.020668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.020562 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497"} err="failed to get container status \"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497\": rpc error: code = NotFound desc = could not find container \"f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497\": container with ID starting with f8d4ea41b69e86a6dcd81ee965ba973ff932ba712a8b06e7ed2fc0e033bc8497 not found: ID does not exist" Nov 03 09:42:50.020668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.020587 4543 scope.go:122] "RemoveContainer" containerID="7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045" Nov 03 09:42:50.020876 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:42:50.020854 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045\": container with ID starting with 7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045 not found: ID does not exist" containerID="7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045" Nov 03 09:42:50.020928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.020888 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045"} err="failed to get container status \"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045\": rpc error: code = NotFound desc = could not find container \"7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045\": container with ID starting with 7034e9758b5d9c0ecff1cfb1cdfb9316999e27a52c744fbd38cbf4a8b7176045 not found: ID does not exist" Nov 03 09:42:50.303226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.284731 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:42:50.309844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.309814 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8204d6-openshift-default-54fdf48699-k7d6w"] Nov 03 09:42:50.451755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.451720 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" path="/var/lib/kubelet/pods/93c422f0-cc6a-4023-9983-ca1fd2f03248/volumes" Nov 03 09:42:50.950557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.950522 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:42:50.950760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.950742 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 09:42:50.950822 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:50.950784 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 09:42:51.949960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:51.949927 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 09:42:51.950431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:51.949990 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 09:42:52.507702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:52.507666 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:52.950312 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:52.950279 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 09:42:52.950750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:52.950329 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 09:42:53.950631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:53.950594 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 09:42:53.951065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:53.950660 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 09:42:54.344752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:54.344671 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:54.950338 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:54.950302 4543 patch_prober.go:29] interesting pod/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 09:42:54.950523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:54.950354 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 09:42:55.248190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:55.248074 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:55.841358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:55.841326 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:56.002217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.001110 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:42:56.004190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.002631 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:42:56.007751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.004404 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:42:56.700367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.700332 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:56.839739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.839706 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:56.975756 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:56.975674 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:58.227988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:58.227920 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:58.364272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:58.364203 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:59.685092 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:59.685049 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:59.817946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:59.817915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:42:59.952098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:42:59.951997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:01.254028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:01.253989 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:04.791825 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:04.791781 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:05.121613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:05.121578 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:05.561125 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:43:05.561045 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:43:09.516955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:09.516915 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:09.746166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:09.746117 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:14.617742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:14.617675 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:14.768045 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:14.767998 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:15.754573 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:15.754542 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:15.898939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:15.898893 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:16.930746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:16.930636 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:20.374217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:20.372176 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:22.244213 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:22.244173 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:22.852123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:22.852084 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:24.835378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:24.835340 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:26.034455 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:26.034422 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:31.159085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:31.159053 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:35.561602 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:43:35.561558 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:43:36.144629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:36.144593 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:37.547173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:37.547114 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:39.556518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:39.556485 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:41.134078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:41.134046 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:44.379992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:44.379956 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:44.821897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:44.821808 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:44.961752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:44.961718 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:46.438753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:46.438720 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:51.156843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:51.156808 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:51.735832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:51.735787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:43:56.981957 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:43:56.981923 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:01.592035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:01.591995 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:01.942686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:01.942652 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:05.467216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:05.467176 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:05.560825 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:05.560790 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:44:06.236243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:06.236208 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:06.941813 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:06.941775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:11.949521 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:11.949490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:12.915239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:12.915206 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:16.939130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:16.939093 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:21.241165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:21.240230 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:44:21.241165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:21.240501 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" podUID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" containerName="maas-api" containerID="cri-o://0397f0131a3c16fb816bcb9558dea4f517192abf9c30ba6b1642c9593f0d99e3" gracePeriod=30 Nov 03 09:44:21.337179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:21.334930 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:22.362166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.361405 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:44:22.602341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.601100 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:44:22.738218 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.718192 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 09:44:22.837442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.824372 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:22.961504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.961470 4543 generic.go:356] "Generic (PLEG): container finished" podID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" containerID="0397f0131a3c16fb816bcb9558dea4f517192abf9c30ba6b1642c9593f0d99e3" exitCode=0 Nov 03 09:44:22.961770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.961518 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" event={"ID":"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31","Type":"ContainerDied","Data":"0397f0131a3c16fb816bcb9558dea4f517192abf9c30ba6b1642c9593f0d99e3"} Nov 03 09:44:22.967285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:22.967261 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:44:23.077417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.077331 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") pod \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " Nov 03 09:44:23.077417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.077401 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\") pod \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " Nov 03 09:44:23.077650 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.077563 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\") pod \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " Nov 03 09:44:23.077720 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.077700 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\" (UniqueName: \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\") pod \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\" (UID: \"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31\") " Nov 03 09:44:23.078179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.078127 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca" pod "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" (UID: "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:44:23.080033 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.080008 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls" pod "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" (UID: "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:44:23.080276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.080179 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs" pod "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" (UID: "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:44:23.081219 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.081193 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw" pod "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" (UID: "61e8d938-bc22-4ee1-b4aa-a4b7fe603b31"). InnerVolumeSpecName "kube-api-access-vtshw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:23.179933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.179892 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:23.179933 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.179933 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtshw\" (UniqueName: \"kubernetes.io/projected/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-kube-api-access-vtshw\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:23.180177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.179948 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:23.180177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.179962 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:23.452996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.452947 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:23.968911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.968883 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" Nov 03 09:44:23.969077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.968885 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z" event={"ID":"61e8d938-bc22-4ee1-b4aa-a4b7fe603b31","Type":"ContainerDied","Data":"0dc630871752cdaa07785addea135a259d7d05b4530130e7ee7d7111129ad84e"} Nov 03 09:44:23.969077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:23.969014 4543 scope.go:122] "RemoveContainer" containerID="0397f0131a3c16fb816bcb9558dea4f517192abf9c30ba6b1642c9593f0d99e3" Nov 03 09:44:24.141545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:24.141512 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:44:24.144237 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:24.144208 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-373116-6cdc89d87b-fwb5z"] Nov 03 09:44:24.452429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:24.452356 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" path="/var/lib/kubelet/pods/61e8d938-bc22-4ee1-b4aa-a4b7fe603b31/volumes" Nov 03 09:44:28.457568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:28.457538 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:29.494518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:29.494475 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:29.634572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:29.634524 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:31.167568 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.167508 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:31.676036 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.676003 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:44:31.790904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.790868 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:44:31.791224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.791194 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" containerID="cri-o://f8dc07b1fe06d60a0ade71464becf2353e8176f060908128adc0ca218f6f0522" gracePeriod=30 Nov 03 09:44:31.806523 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.806482 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:31.965398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:31.965316 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:32.105708 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:32.105671 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:32.250965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:32.250874 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:32.586215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:32.585613 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:34.034885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:34.034824 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:35.560357 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:35.560323 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:44:35.711317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:35.711260 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:35.962374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:35.962340 4543 patch_prober.go:29] interesting pod/e2e-extmod-a-373116-openshift-default-559449785-2jxqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:44:35.962589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:35.962415 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:44:36.575264 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:36.575229 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-05110bbd79d3: secret "agc-cred-05110bbd79d3" not found Nov 03 09:44:36.575264 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:36.575253 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-546e9f5ec78c: secret "agc-cred-546e9f5ec78c" not found Nov 03 09:44:36.575264 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:36.575259 4543 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1c3ed9d233a: secret "agc-cred-b1c3ed9d233a" not found Nov 03 09:44:36.575264 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:36.575268 4543 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8: [secret "agc-cred-05110bbd79d3" not found, secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:44:36.575773 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:36.575330 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials podName:996396de-ccf4-44be-90c1-ac25611c3505 nodeName:}" failed. No retries permitted until 2026-11-03 09:46:38.575316687 +0000 UTC m=+4022.243357428 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials") pod "payload-processing-external-model-5c4fddf7b5-kmvr8" (UID: "996396de-ccf4-44be-90c1-ac25611c3505") : [secret "agc-cred-05110bbd79d3" not found, secret "agc-cred-546e9f5ec78c" not found, secret "agc-cred-b1c3ed9d233a" not found] Nov 03 09:44:36.685200 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:36.678336 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" podUID="996396de-ccf4-44be-90c1-ac25611c3505" containerName="payload-processing" containerID="cri-o://b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069" gracePeriod=25 Nov 03 09:44:36.999181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:36.981778 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:44:37.029118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.029084 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:44:37.056784 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.056753 4543 generic.go:356] "Generic (PLEG): container finished" podID="996396de-ccf4-44be-90c1-ac25611c3505" containerID="b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069" exitCode=0 Nov 03 09:44:37.056966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.056856 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" Nov 03 09:44:37.057040 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.056975 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" event={"ID":"996396de-ccf4-44be-90c1-ac25611c3505","Type":"ContainerDied","Data":"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069"} Nov 03 09:44:37.057085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.057075 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8" event={"ID":"996396de-ccf4-44be-90c1-ac25611c3505","Type":"ContainerDied","Data":"3ace3157f65770d2eb9c29bc653572f37e749f1c87c9cdc2d9ecc983d74cc95f"} Nov 03 09:44:37.057129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.057106 4543 scope.go:122] "RemoveContainer" containerID="b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069" Nov 03 09:44:37.061280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.060122 4543 generic.go:356] "Generic (PLEG): container finished" podID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerID="f8dc07b1fe06d60a0ade71464becf2353e8176f060908128adc0ca218f6f0522" exitCode=0 Nov 03 09:44:37.061280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.060178 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerDied","Data":"f8dc07b1fe06d60a0ade71464becf2353e8176f060908128adc0ca218f6f0522"} Nov 03 09:44:37.077303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.077286 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:44:37.081777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.081760 4543 scope.go:122] "RemoveContainer" containerID="b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069" Nov 03 09:44:37.082087 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:44:37.082066 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069\": container with ID starting with b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069 not found: ID does not exist" containerID="b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069" Nov 03 09:44:37.082211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.082100 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069"} err="failed to get container status \"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069\": rpc error: code = NotFound desc = could not find container \"b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069\": container with ID starting with b4e8892bdc46666e6bca5a771fb2d88939d03d10ddd4852e69e6f24b47f7b069 not found: ID does not exist" Nov 03 09:44:37.082211 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.082116 4543 scope.go:122] "RemoveContainer" containerID="7a4f81a92dec70e37d8774bb78fc309c559ca25568e51d5105fc2bc2e30b654e" Nov 03 09:44:37.085005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.084986 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:44:37.118329 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118305 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\") pod \"996396de-ccf4-44be-90c1-ac25611c3505\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " Nov 03 09:44:37.118493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118366 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\") pod \"996396de-ccf4-44be-90c1-ac25611c3505\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " Nov 03 09:44:37.118493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\") pod \"996396de-ccf4-44be-90c1-ac25611c3505\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " Nov 03 09:44:37.118623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118531 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\") pod \"996396de-ccf4-44be-90c1-ac25611c3505\" (UID: \"996396de-ccf4-44be-90c1-ac25611c3505\") " Nov 03 09:44:37.118747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118721 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume" pod "996396de-ccf4-44be-90c1-ac25611c3505" (UID: "996396de-ccf4-44be-90c1-ac25611c3505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:44:37.119001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.118979 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume" pod "996396de-ccf4-44be-90c1-ac25611c3505" (UID: "996396de-ccf4-44be-90c1-ac25611c3505"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:44:37.119316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.119219 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.119316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.119265 4543 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/996396de-ccf4-44be-90c1-ac25611c3505-routing-overlay-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.120877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.120854 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials" pod "996396de-ccf4-44be-90c1-ac25611c3505" (UID: "996396de-ccf4-44be-90c1-ac25611c3505"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:37.121310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.121268 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw" pod "996396de-ccf4-44be-90c1-ac25611c3505" (UID: "996396de-ccf4-44be-90c1-ac25611c3505"). InnerVolumeSpecName "kube-api-access-nprqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:37.219685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219650 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.219854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219699 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.219854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219760 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.219854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219776 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.219854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219828 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.220078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219894 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.220078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.219947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.220209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220091 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.220209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220185 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:44:37.220209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220204 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\") pod \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\" (UID: \"2c46e78a-dc06-463a-8e74-7bd5f4cb0381\") " Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220595 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220841 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220861 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprqw\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-kube-api-access-nprqw\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220877 4543 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/996396de-ccf4-44be-90c1-ac25611c3505-provider-credentials\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220890 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.220955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.220903 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.221448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.221336 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:44:37.221448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.221392 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:44:37.222501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.222451 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:44:37.222948 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.222921 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "kube-api-access-flxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:37.223853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.223828 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:44:37.223943 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.223864 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token" pod "2c46e78a-dc06-463a-8e74-7bd5f4cb0381" (UID: "2c46e78a-dc06-463a-8e74-7bd5f4cb0381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:44:37.322517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322487 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxrf\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-kube-api-access-flxrf\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.322517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322511 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.322517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322522 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.322744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322530 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.322744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322538 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.322744 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.322546 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c46e78a-dc06-463a-8e74-7bd5f4cb0381-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:44:37.480719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.480640 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:44:37.485429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:37.485403 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c4fddf7b5-kmvr8"] Nov 03 09:44:38.067817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.067738 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" Nov 03 09:44:38.067817 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.067760 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb" event={"ID":"2c46e78a-dc06-463a-8e74-7bd5f4cb0381","Type":"ContainerDied","Data":"b54c40ffcf6e186b61d03c3ea5e793cdb1ff39d385c28c47a052fbc426ef4c38"} Nov 03 09:44:38.068345 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.067823 4543 scope.go:122] "RemoveContainer" containerID="f8dc07b1fe06d60a0ade71464becf2353e8176f060908128adc0ca218f6f0522" Nov 03 09:44:38.175950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.175891 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:38.305045 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.305016 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:44:38.309979 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.309939 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-373116-openshift-default-559449785-2jxqb"] Nov 03 09:44:38.312335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.312309 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:38.452621 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.452580 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" path="/var/lib/kubelet/pods/2c46e78a-dc06-463a-8e74-7bd5f4cb0381/volumes" Nov 03 09:44:38.453090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:38.453070 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996396de-ccf4-44be-90c1-ac25611c3505" path="/var/lib/kubelet/pods/996396de-ccf4-44be-90c1-ac25611c3505/volumes" Nov 03 09:44:39.240446 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:39.240412 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:39.712446 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:39.712411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:42.769776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:42.769743 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:42.889567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:42.889533 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:43.482837 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:43.482800 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:43.911382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:43.911344 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:44.325220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:44.325094 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:44.464590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:44.464560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:44.602530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:44.602450 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:44.803010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:44.802978 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:44.951031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:44.950997 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:45.898525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:45.898490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:51.159080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:51.159000 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:51.925294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:51.925261 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:55.061043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:55.061007 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:55.206317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:55.206286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:56.392560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:56.392521 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:56.659386 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:56.659308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:57.172946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:57.172913 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:58.689604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:58.689572 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:44:58.824946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:44:58.824912 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:00.213250 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.213211 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:00.213989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.213967 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.213990 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214010 4543 cpu_manager.go:414] "containerMap: removing container" podUID="967c1242-1899-4a31-9d42-3ce0fe87dc79" containerName="maas-api" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214017 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967c1242-1899-4a31-9d42-3ce0fe87dc79" containerName="maas-api" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214038 4543 cpu_manager.go:414] "containerMap: removing container" podUID="996396de-ccf4-44be-90c1-ac25611c3505" containerName="payload-processing" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214047 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996396de-ccf4-44be-90c1-ac25611c3505" containerName="payload-processing" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214059 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214066 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214078 4543 cpu_manager.go:414] "containerMap: removing container" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214084 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214096 4543 cpu_manager.go:414] "containerMap: removing container" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214104 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214115 4543 cpu_manager.go:414] "containerMap: removing container" podUID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" containerName="maas-api" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214122 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" containerName="maas-api" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214292 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="996396de-ccf4-44be-90c1-ac25611c3505" containerName="payload-processing" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214308 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214319 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c422f0-cc6a-4023-9983-ca1fd2f03248" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214329 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214337 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e8d938-bc22-4ee1-b4aa-a4b7fe603b31" containerName="maas-api" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214354 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c46e78a-dc06-463a-8e74-7bd5f4cb0381" containerName="istio-proxy" Nov 03 09:45:00.214611 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.214364 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="967c1242-1899-4a31-9d42-3ce0fe87dc79" containerName="maas-api" Nov 03 09:45:00.218107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.218088 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.218781 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.218760 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:00.222482 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.222465 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.327375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.327346 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\"" Nov 03 09:45:00.340103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.340068 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtp8\" (UniqueName: \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.340291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.340110 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.340368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.340295 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62f4l\" (UniqueName: \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.340368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.340352 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.387868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.387832 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:00.441738 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.441701 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.441962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.441842 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtp8\" (UniqueName: \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.441962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.441870 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.442164 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.441992 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62f4l\" (UniqueName: \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.442857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.442831 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.443196 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.443126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.453628 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.453605 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:00.502540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.502471 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62f4l\" (UniqueName: \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.504235 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.504214 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtp8\" (UniqueName: \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.532296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.532273 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:00.538925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.538906 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:00.911885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.911860 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:00.913849 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:45:00.913825 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5ffa15_de72_4ca1_a28b_ed9c19c7ee31.slice/crio-f569d07cf4ef840621702b78bba3b76999bbbddc93368575a4bfb6eec70dafc2 WatchSource:0}: Error finding container f569d07cf4ef840621702b78bba3b76999bbbddc93368575a4bfb6eec70dafc2: Status 404 returned error can't find the container with id f569d07cf4ef840621702b78bba3b76999bbbddc93368575a4bfb6eec70dafc2 Nov 03 09:45:00.964467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:00.964445 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:00.964896 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:45:00.964871 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c149402_cbec_4cd9_ae0f_9f13b55717d0.slice/crio-451ae31fe1c926a21123797fe78e1f96353da59cf2c412e84e3dee1a04e449e7 WatchSource:0}: Error finding container 451ae31fe1c926a21123797fe78e1f96353da59cf2c412e84e3dee1a04e449e7: Status 404 returned error can't find the container with id 451ae31fe1c926a21123797fe78e1f96353da59cf2c412e84e3dee1a04e449e7 Nov 03 09:45:01.314887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.314860 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:45:01.315322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.315053 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" podUID="8a44fe82-9028-4606-9c20-70f308e048b7" containerName="maas-api" containerID="cri-o://489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85" gracePeriod=30 Nov 03 09:45:01.323019 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.322986 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerStarted","Data":"ee96c38d7de652f72583176ad5dd15319a74c16c730fad94880bba6acd4e3f4b"} Nov 03 09:45:01.323212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.323049 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerStarted","Data":"f569d07cf4ef840621702b78bba3b76999bbbddc93368575a4bfb6eec70dafc2"} Nov 03 09:45:01.324658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.324624 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" event={"ID":"8c149402-cbec-4cd9-ae0f-9f13b55717d0","Type":"ContainerStarted","Data":"451ae31fe1c926a21123797fe78e1f96353da59cf2c412e84e3dee1a04e449e7"} Nov 03 09:45:01.411296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.411267 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:01.546670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.546607 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" podStartSLOduration=1.5465862559999999 podStartE2EDuration="1.546586256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:01.545496929 +0000 UTC m=+3925.213537667" watchObservedRunningTime="2026-11-03 09:45:01.546586256 +0000 UTC m=+3925.214626999" Nov 03 09:45:01.656846 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.656817 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:45:01.758305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758222 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\") pod \"8a44fe82-9028-4606-9c20-70f308e048b7\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " Nov 03 09:45:01.758305 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758276 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\" (UniqueName: \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\") pod \"8a44fe82-9028-4606-9c20-70f308e048b7\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " Nov 03 09:45:01.758488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758331 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\") pod \"8a44fe82-9028-4606-9c20-70f308e048b7\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " Nov 03 09:45:01.758488 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758415 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\") pod \"8a44fe82-9028-4606-9c20-70f308e048b7\" (UID: \"8a44fe82-9028-4606-9c20-70f308e048b7\") " Nov 03 09:45:01.758602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758580 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca" pod "8a44fe82-9028-4606-9c20-70f308e048b7" (UID: "8a44fe82-9028-4606-9c20-70f308e048b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:01.758922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.758896 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a44fe82-9028-4606-9c20-70f308e048b7-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:01.760585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.760563 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs" pod "8a44fe82-9028-4606-9c20-70f308e048b7" (UID: "8a44fe82-9028-4606-9c20-70f308e048b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:45:01.760585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.760579 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls" pod "8a44fe82-9028-4606-9c20-70f308e048b7" (UID: "8a44fe82-9028-4606-9c20-70f308e048b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:45:01.760716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.760590 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk" pod "8a44fe82-9028-4606-9c20-70f308e048b7" (UID: "8a44fe82-9028-4606-9c20-70f308e048b7"). InnerVolumeSpecName "kube-api-access-xm6kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:01.860716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.860681 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:01.860716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.860710 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a44fe82-9028-4606-9c20-70f308e048b7-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:01.860716 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.860720 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6kk\" (UniqueName: \"kubernetes.io/projected/8a44fe82-9028-4606-9c20-70f308e048b7-kube-api-access-xm6kk\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:01.947557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:01.947513 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:02.341564 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.341529 4543 generic.go:356] "Generic (PLEG): container finished" podID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerID="25fa4cd09e3ad512e83e93c8cd665a0b88d2e6f660cd0ed47c7eac58535a0daa" exitCode=60 Nov 03 09:45:02.342038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.341606 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" event={"ID":"8c149402-cbec-4cd9-ae0f-9f13b55717d0","Type":"ContainerDied","Data":"25fa4cd09e3ad512e83e93c8cd665a0b88d2e6f660cd0ed47c7eac58535a0daa"} Nov 03 09:45:02.342038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.341852 4543 scope.go:122] "RemoveContainer" containerID="25fa4cd09e3ad512e83e93c8cd665a0b88d2e6f660cd0ed47c7eac58535a0daa" Nov 03 09:45:02.343554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.343435 4543 generic.go:356] "Generic (PLEG): container finished" podID="8a44fe82-9028-4606-9c20-70f308e048b7" containerID="489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85" exitCode=0 Nov 03 09:45:02.343554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.343515 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" event={"ID":"8a44fe82-9028-4606-9c20-70f308e048b7","Type":"ContainerDied","Data":"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85"} Nov 03 09:45:02.343670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.343576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" event={"ID":"8a44fe82-9028-4606-9c20-70f308e048b7","Type":"ContainerDied","Data":"ea49d64e1c798d02424d62f51b0bdd77502c84499057bf0115bb664fca0dd2af"} Nov 03 09:45:02.343670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.343604 4543 scope.go:122] "RemoveContainer" containerID="489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85" Nov 03 09:45:02.343849 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.343827 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8" Nov 03 09:45:02.345433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.345343 4543 generic.go:356] "Generic (PLEG): container finished" podID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerID="ee96c38d7de652f72583176ad5dd15319a74c16c730fad94880bba6acd4e3f4b" exitCode=60 Nov 03 09:45:02.345433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.345393 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerDied","Data":"ee96c38d7de652f72583176ad5dd15319a74c16c730fad94880bba6acd4e3f4b"} Nov 03 09:45:02.345704 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.345688 4543 scope.go:122] "RemoveContainer" containerID="ee96c38d7de652f72583176ad5dd15319a74c16c730fad94880bba6acd4e3f4b" Nov 03 09:45:02.356610 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.356589 4543 scope.go:122] "RemoveContainer" containerID="489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85" Nov 03 09:45:02.357448 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:02.357418 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85\": container with ID starting with 489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85 not found: ID does not exist" containerID="489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85" Nov 03 09:45:02.357528 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.357457 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85"} err="failed to get container status \"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85\": rpc error: code = NotFound desc = could not find container \"489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85\": container with ID starting with 489cee1c5663a7d7cfdd7f839165180b5a51c40193c165dbf36f6c2331b84d85 not found: ID does not exist" Nov 03 09:45:02.588722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.588689 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:45:02.595783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:02.595759 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-872281-7c74d59bc7-qkzz8"] Nov 03 09:45:03.356456 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.356421 4543 generic.go:356] "Generic (PLEG): container finished" podID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerID="cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728" exitCode=60 Nov 03 09:45:03.356969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.356533 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" event={"ID":"8c149402-cbec-4cd9-ae0f-9f13b55717d0","Type":"ContainerDied","Data":"cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728"} Nov 03 09:45:03.356969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.356579 4543 scope.go:122] "RemoveContainer" containerID="25fa4cd09e3ad512e83e93c8cd665a0b88d2e6f660cd0ed47c7eac58535a0daa" Nov 03 09:45:03.356969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.356960 4543 scope.go:122] "RemoveContainer" containerID="cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728" Nov 03 09:45:03.357266 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:03.357248 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l_odh-ai-gateway-infra(8c149402-cbec-4cd9-ae0f-9f13b55717d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" Nov 03 09:45:03.361407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.361378 4543 generic.go:356] "Generic (PLEG): container finished" podID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerID="f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217" exitCode=60 Nov 03 09:45:03.361558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.361430 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerDied","Data":"f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217"} Nov 03 09:45:03.361892 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.361877 4543 scope.go:122] "RemoveContainer" containerID="f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217" Nov 03 09:45:03.362194 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:03.362134 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84_odh-ai-gateway-infra(ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" Nov 03 09:45:03.363157 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.363122 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:03.370474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.370460 4543 scope.go:122] "RemoveContainer" containerID="ee96c38d7de652f72583176ad5dd15319a74c16c730fad94880bba6acd4e3f4b" Nov 03 09:45:03.531707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.530960 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:03.703665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:03.703631 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:04.369366 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:04.369337 4543 scope.go:122] "RemoveContainer" containerID="cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728" Nov 03 09:45:04.369815 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:04.369638 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l_odh-ai-gateway-infra(8c149402-cbec-4cd9-ae0f-9f13b55717d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" Nov 03 09:45:04.370949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:04.370933 4543 scope.go:122] "RemoveContainer" containerID="f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217" Nov 03 09:45:04.371213 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:04.371189 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84_odh-ai-gateway-infra(ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" Nov 03 09:45:04.452110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:04.452067 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a44fe82-9028-4606-9c20-70f308e048b7" path="/var/lib/kubelet/pods/8a44fe82-9028-4606-9c20-70f308e048b7/volumes" Nov 03 09:45:05.276936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:05.276899 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:05.405304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:05.405263 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:05.563679 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:05.563584 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:45:05.692478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:05.692446 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:05.898310 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:05.898279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:07.367103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:07.367074 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:07.507754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:07.507719 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:08.527464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:08.527434 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:08.708340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:08.708303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:08.899406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:08.899367 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:09.030220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:09.030189 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:09.230633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:09.230551 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:09.721747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:09.721718 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:45:09.722298 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:09.721951 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" containerID="cri-o://0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1" gracePeriod=30 Nov 03 09:45:09.722604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:09.722461 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:10.418515 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:10.418480 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:14.294643 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:14.294610 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:14.451477 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:14.451443 4543 scope.go:122] "RemoveContainer" containerID="f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217" Nov 03 09:45:14.988843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:14.988817 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:45:15.102835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.102795 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.102847 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.102904 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.102979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103052 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103084 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:15.103178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103097 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103241 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:15.103359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103277 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:15.103359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103284 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103319 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:15.103575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103419 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:15.103575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103323 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103575 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103519 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\") pod \"87449f85-afd4-47a1-ad8d-bee41013ba65\" (UID: \"87449f85-afd4-47a1-ad8d-bee41013ba65\") " Nov 03 09:45:15.103949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103929 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.103949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103951 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87449f85-afd4-47a1-ad8d-bee41013ba65-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.104125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103964 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.104125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103975 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.104125 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.103986 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.105453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.105430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:45:15.105550 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.105499 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "kube-api-access-dwg6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:15.105909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.105889 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:15.105976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.105893 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo" pod "87449f85-afd4-47a1-ad8d-bee41013ba65" (UID: "87449f85-afd4-47a1-ad8d-bee41013ba65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:45:15.205478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.205446 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.205478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.205475 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.205478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.205484 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwg6w\" (UniqueName: \"kubernetes.io/projected/87449f85-afd4-47a1-ad8d-bee41013ba65-kube-api-access-dwg6w\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.205715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.205493 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87449f85-afd4-47a1-ad8d-bee41013ba65-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:15.438331 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.438297 4543 generic.go:356] "Generic (PLEG): container finished" podID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerID="f3c3ff1afcf739d2efa3a11361e1a75e23a799816e6ca5ae50fad6576bb689dc" exitCode=60 Nov 03 09:45:15.438793 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.438371 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerDied","Data":"f3c3ff1afcf739d2efa3a11361e1a75e23a799816e6ca5ae50fad6576bb689dc"} Nov 03 09:45:15.438793 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.438429 4543 scope.go:122] "RemoveContainer" containerID="f176e0bef4b952aff5f665085e395eaef607cd6724965850acefdb6b168bc217" Nov 03 09:45:15.438910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.438845 4543 scope.go:122] "RemoveContainer" containerID="f3c3ff1afcf739d2efa3a11361e1a75e23a799816e6ca5ae50fad6576bb689dc" Nov 03 09:45:15.439182 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:15.439152 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84_odh-ai-gateway-infra(ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" Nov 03 09:45:15.440223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.440203 4543 generic.go:356] "Generic (PLEG): container finished" podID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerID="0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1" exitCode=0 Nov 03 09:45:15.440318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.440250 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" Nov 03 09:45:15.440318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.440287 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerDied","Data":"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1"} Nov 03 09:45:15.440392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.440328 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n" event={"ID":"87449f85-afd4-47a1-ad8d-bee41013ba65","Type":"ContainerDied","Data":"74cbe93d00bc1f3c29761c58934d7d503a63d8548e2332587b436d58018431e5"} Nov 03 09:45:15.450996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.450980 4543 scope.go:122] "RemoveContainer" containerID="0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1" Nov 03 09:45:15.459973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.459795 4543 scope.go:122] "RemoveContainer" containerID="581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454" Nov 03 09:45:15.469829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.469806 4543 scope.go:122] "RemoveContainer" containerID="0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1" Nov 03 09:45:15.470115 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:15.470089 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1\": container with ID starting with 0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1 not found: ID does not exist" containerID="0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1" Nov 03 09:45:15.470242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.470123 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1"} err="failed to get container status \"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1\": rpc error: code = NotFound desc = could not find container \"0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1\": container with ID starting with 0eb7c5554aa3bdebc62efab46c9fda9053851daca2c4387cd103ef8a0e69eca1 not found: ID does not exist" Nov 03 09:45:15.470242 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.470154 4543 scope.go:122] "RemoveContainer" containerID="581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454" Nov 03 09:45:15.470377 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:15.470360 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454\": container with ID starting with 581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454 not found: ID does not exist" containerID="581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454" Nov 03 09:45:15.470428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.470381 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454"} err="failed to get container status \"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454\": rpc error: code = NotFound desc = could not find container \"581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454\": container with ID starting with 581dfb87d4339436052dfef950752e6ab9c5e9391a978e880221c3b298efa454 not found: ID does not exist" Nov 03 09:45:15.671458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.671420 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:45:15.721967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:15.721934 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872281-openshift-default-79bd6c9f9f-hb22n"] Nov 03 09:45:16.238841 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.238806 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:16.378944 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.378912 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:16.452537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.452502 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" path="/var/lib/kubelet/pods/87449f85-afd4-47a1-ad8d-bee41013ba65/volumes" Nov 03 09:45:16.565954 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.565871 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:16.717648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.717621 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:16.825458 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.825375 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\" (UniqueName: \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\") pod \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " Nov 03 09:45:16.825641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.825516 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\") pod \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\" (UID: \"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31\") " Nov 03 09:45:16.825869 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.825845 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca" pod "ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" (UID: "ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:16.826117 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.826099 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-maas-api-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:16.827761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.827742 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8" pod "ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" (UID: "ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31"). InnerVolumeSpecName "kube-api-access-9gtp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:16.927579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:16.927547 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gtp8\" (UniqueName: \"kubernetes.io/projected/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31-kube-api-access-9gtp8\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:17.444614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.444585 4543 scope.go:122] "RemoveContainer" containerID="cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728" Nov 03 09:45:17.470404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.470376 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" Nov 03 09:45:17.470785 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.470392 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84" event={"ID":"ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31","Type":"ContainerDied","Data":"f569d07cf4ef840621702b78bba3b76999bbbddc93368575a4bfb6eec70dafc2"} Nov 03 09:45:17.470785 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.470472 4543 scope.go:122] "RemoveContainer" containerID="f3c3ff1afcf739d2efa3a11361e1a75e23a799816e6ca5ae50fad6576bb689dc" Nov 03 09:45:17.487235 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.487204 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:17.636651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.636610 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:17.642666 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.642638 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:17.651475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:17.651446 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894985-9nf84"] Nov 03 09:45:18.453294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.453259 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" path="/var/lib/kubelet/pods/ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31/volumes" Nov 03 09:45:18.478596 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.478570 4543 generic.go:356] "Generic (PLEG): container finished" podID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerID="221ebaf75edc3176a33e3a81901cd4657e26afb947a43d31c8609a26528d78c9" exitCode=60 Nov 03 09:45:18.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.478642 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" event={"ID":"8c149402-cbec-4cd9-ae0f-9f13b55717d0","Type":"ContainerDied","Data":"221ebaf75edc3176a33e3a81901cd4657e26afb947a43d31c8609a26528d78c9"} Nov 03 09:45:18.479001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.478699 4543 scope.go:122] "RemoveContainer" containerID="cac6d4127694feee9801484786f9a54d97a06070f3c25b131c32acc5e8814728" Nov 03 09:45:18.479084 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.479013 4543 scope.go:122] "RemoveContainer" containerID="221ebaf75edc3176a33e3a81901cd4657e26afb947a43d31c8609a26528d78c9" Nov 03 09:45:18.479399 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:18.479367 4543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l_odh-ai-gateway-infra(8c149402-cbec-4cd9-ae0f-9f13b55717d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" Nov 03 09:45:18.746238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.746121 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:18.918878 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:18.918818 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:19.173761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.173717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:19.607949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.607860 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:19.773609 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.773575 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:19.813511 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.813471 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:19.870988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.870951 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\" (UniqueName: \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\") pod \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " Nov 03 09:45:19.871296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.871275 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\") pod \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\" (UID: \"8c149402-cbec-4cd9-ae0f-9f13b55717d0\") " Nov 03 09:45:19.872088 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.872062 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca" pod "8c149402-cbec-4cd9-ae0f-9f13b55717d0" (UID: "8c149402-cbec-4cd9-ae0f-9f13b55717d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:45:19.878006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.877973 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l" pod "8c149402-cbec-4cd9-ae0f-9f13b55717d0" (UID: "8c149402-cbec-4cd9-ae0f-9f13b55717d0"). InnerVolumeSpecName "kube-api-access-62f4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:45:19.972457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.972423 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c149402-cbec-4cd9-ae0f-9f13b55717d0-maas-api-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:19.972457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:19.972448 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62f4l\" (UniqueName: \"kubernetes.io/projected/8c149402-cbec-4cd9-ae0f-9f13b55717d0-kube-api-access-62f4l\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:45:20.494518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:20.494492 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" Nov 03 09:45:20.494518 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:20.494506 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l" event={"ID":"8c149402-cbec-4cd9-ae0f-9f13b55717d0","Type":"ContainerDied","Data":"451ae31fe1c926a21123797fe78e1f96353da59cf2c412e84e3dee1a04e449e7"} Nov 03 09:45:20.494774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:20.494575 4543 scope.go:122] "RemoveContainer" containerID="221ebaf75edc3176a33e3a81901cd4657e26afb947a43d31c8609a26528d78c9" Nov 03 09:45:20.663061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:20.663032 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:20.715239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:20.715133 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894985-7rs2l"] Nov 03 09:45:22.451762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:22.451727 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" path="/var/lib/kubelet/pods/8c149402-cbec-4cd9-ae0f-9f13b55717d0/volumes" Nov 03 09:45:23.039373 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:23.039346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:24.845723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:24.845696 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:25.458927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:25.458893 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:29.842524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:29.842488 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:31.622380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:31.622344 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:34.441165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.440546 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441428 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441448 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441467 4543 cpu_manager.go:414] "containerMap: removing container" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441475 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441495 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441502 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441518 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441526 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441541 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441547 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441591 4543 cpu_manager.go:414] "containerMap: removing container" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441599 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441627 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8a44fe82-9028-4606-9c20-70f308e048b7" containerName="maas-api" Nov 03 09:45:34.441655 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441635 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a44fe82-9028-4606-9c20-70f308e048b7" containerName="maas-api" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441753 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a44fe82-9028-4606-9c20-70f308e048b7" containerName="maas-api" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441766 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441776 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441787 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441795 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441804 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.442460 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.441819 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:45:34.486548 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.486522 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.523875 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.523846 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:45:34.555489 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.555458 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 09:45:34.555775 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.555759 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 09:45:34.556026 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.556007 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9gqlj\"" Nov 03 09:45:34.556226 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.556210 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 09:45:34.640111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.640032 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.640471 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.640451 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.640679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.640662 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.640855 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.640839 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.641013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.641000 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.641207 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.641191 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.641388 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.641371 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2cws\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743683 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743793 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743811 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743833 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743855 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743884 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.745276 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.743926 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2cws\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.747107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.747084 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.753437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.753384 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.757915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.757850 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.758547 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.758529 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.766988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.766960 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.803808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.803769 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2cws\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.805184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.805156 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-knkzg\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:34.816907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:34.816884 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:35.059291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.059206 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:35.250703 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.250682 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:45:35.251425 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:45:35.251400 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faebc12_103c_445c_9f2f_449ecb1e23cf.slice/crio-7f948f830028551035a12aec79917158829d2fd0b2668dc592974dd409ec0e1a WatchSource:0}: Error finding container 7f948f830028551035a12aec79917158829d2fd0b2668dc592974dd409ec0e1a: Status 404 returned error can't find the container with id 7f948f830028551035a12aec79917158829d2fd0b2668dc592974dd409ec0e1a Nov 03 09:45:35.253787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.253754 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:45:35.253870 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.253815 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:45:35.564709 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:45:35.564603 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:45:35.657684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.657646 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" event={"ID":"1faebc12-103c-445c-9f2f-449ecb1e23cf","Type":"ContainerStarted","Data":"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0"} Nov 03 09:45:35.657900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.657701 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" event={"ID":"1faebc12-103c-445c-9f2f-449ecb1e23cf","Type":"ContainerStarted","Data":"7f948f830028551035a12aec79917158829d2fd0b2668dc592974dd409ec0e1a"} Nov 03 09:45:35.657900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.657751 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:35.780668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:35.780600 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" podStartSLOduration=1.7805815329999999 podStartE2EDuration="1.780581533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:35.779700778 +0000 UTC m=+3959.447741516" watchObservedRunningTime="2026-11-03 09:45:35.780581533 +0000 UTC m=+3959.448622274" Nov 03 09:45:36.665962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:36.665934 4543 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-knkzg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:45:36.666380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:36.665994 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:45:36.706018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:36.705976 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:36.952813 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:36.952705 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:37.100819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:37.100783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:39.386453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:39.386419 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:39.664852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:39.664781 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:45:40.144282 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:40.144248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:45.156924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:45.156883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:47.610098 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:47.610063 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:50.145425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:50.145387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:52.216443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:52.216413 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:52.808951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:52.808919 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:55.134475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:55.134440 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:45:55.510048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:45:55.509978 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:00.227392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:00.227360 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:02.295943 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:02.295885 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:05.174333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:05.174298 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:05.560567 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:46:05.560476 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:46:06.170639 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:06.170609 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:10.167167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:10.167121 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:13.852166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:13.849741 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:15.233177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:15.233124 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:20.233273 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:20.233191 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:20.993854 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:20.993825 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:23.387803 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:23.387769 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:24.664900 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:24.664866 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:26.476644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:26.476605 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:29.784935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:29.784903 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:29.914431 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:29.914394 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:31.400918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:31.400885 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:31.665557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:31.665482 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:31.907670 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:31.907635 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:31.915773 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:31.915707 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:46:31.915930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:31.915909 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" containerID="cri-o://e7510b8e500653432090b56809bfaaf78fde6a7783c9e94701d7e53103bdb8f4" gracePeriod=30 Nov 03 09:46:35.561447 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:46:35.561410 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:46:35.682355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:35.682320 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:37.030167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.030115 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:37.070816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.070759 4543 generic.go:356] "Generic (PLEG): container finished" podID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerID="e7510b8e500653432090b56809bfaaf78fde6a7783c9e94701d7e53103bdb8f4" exitCode=0 Nov 03 09:46:37.070991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.070825 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerDied","Data":"e7510b8e500653432090b56809bfaaf78fde6a7783c9e94701d7e53103bdb8f4"} Nov 03 09:46:37.070991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.070912 4543 scope.go:122] "RemoveContainer" containerID="892d149c49ab144056187de6d49edcfb97dc3b3878e95934da1cb80cbc5ae961" Nov 03 09:46:37.206223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.206200 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:46:37.351752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.351715 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.351930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.351863 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.351988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.351964 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.352039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352020 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.352173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352106 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.352281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352188 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.352702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352664 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:46:37.352915 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352889 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:46:37.353016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352968 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:46:37.353016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.352222 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.353107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353065 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.353202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353127 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\") pod \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\" (UID: \"8df46ae1-d457-4d77-a6ae-c5c7949309e4\") " Nov 03 09:46:37.353335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353315 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:46:37.353503 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353485 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:46:37.353725 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353708 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.353806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353731 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.353806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353744 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.353806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353758 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.353806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.353771 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.354717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.354683 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:46:37.354831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.354774 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:46:37.354899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.354883 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "kube-api-access-4jfm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:46:37.354972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.354955 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo" pod "8df46ae1-d457-4d77-a6ae-c5c7949309e4" (UID: "8df46ae1-d457-4d77-a6ae-c5c7949309e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:46:37.455082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.455011 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.455082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.455035 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jfm6\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-kube-api-access-4jfm6\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.455082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.455046 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:37.455082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:37.455054 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df46ae1-d457-4d77-a6ae-c5c7949309e4-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:46:38.078368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.078336 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" Nov 03 09:46:38.078368 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.078357 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8" event={"ID":"8df46ae1-d457-4d77-a6ae-c5c7949309e4","Type":"ContainerDied","Data":"77eb03a2d2c68b6971fe7501ca838b491b6fab3a45e340b7451f4bfda7f6711d"} Nov 03 09:46:38.078908 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.078440 4543 scope.go:122] "RemoveContainer" containerID="e7510b8e500653432090b56809bfaaf78fde6a7783c9e94701d7e53103bdb8f4" Nov 03 09:46:38.249585 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.249551 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:46:38.299743 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.299711 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-af514d-openshift-default-657b774ff-vvgt8"] Nov 03 09:46:38.452989 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:38.452954 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" path="/var/lib/kubelet/pods/8df46ae1-d457-4d77-a6ae-c5c7949309e4/volumes" Nov 03 09:46:42.037005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:42.036976 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:47.021087 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:47.021056 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:52.048053 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:52.048015 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:46:57.079174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:46:57.079101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:02.054579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:02.054535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:02.635998 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:02.635958 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:05.560880 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:47:05.560839 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:47:06.506086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:06.506048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:07.043742 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:07.043691 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:12.138417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:12.138378 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:13.251355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:13.251301 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:17.140121 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:17.140089 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:20.479858 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:20.479829 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:22.233878 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:22.233839 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:23.735758 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:23.735679 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:27.236202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:27.236124 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:32.226599 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:32.226566 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:35.560590 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:47:35.560552 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:47:37.250029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:37.249990 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:42.325802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:42.325768 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:43.919002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:43.918971 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:47.298560 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:47.298487 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:52.306470 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:52.306429 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:47:57.226148 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:47:57.226105 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:02.262177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:02.262115 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:02.976357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:02.976321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:05.560958 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:48:05.560920 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:48:06.849589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:06.849541 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:07.214662 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:07.214634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:12.180559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:12.180527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:16.177889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:16.177856 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:17.121635 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:17.121601 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:22.074056 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:22.074026 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:23.419499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:23.419458 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:24.074984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:24.074949 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:25.616343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:25.616311 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:25.773699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:25.773665 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:27.089101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:27.089058 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:32.071995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:32.071955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:35.562081 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:48:35.562045 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:48:37.045925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:37.045890 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:39.965866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:39.965836 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:42.055384 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:42.055353 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:43.709723 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:43.709691 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:43.871760 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:43.871727 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:44.276359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:44.276327 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:47.141433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:47.141398 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:51.087419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:51.087373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:51.224729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:51.224698 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:51.542474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:51.542443 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:52.233604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:52.233573 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:53.118788 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:53.118758 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:57.285513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:57.285480 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:58.209437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:58.209408 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:48:58.337963 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:48:58.337934 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:02.629104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:02.629068 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:04.997295 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:04.997266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:05.560529 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:49:05.560489 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:49:07.201185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:07.201132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:07.591249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:07.591170 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:12.547877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:12.547845 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:14.525767 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:14.525730 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:17.486616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:17.486520 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:22.441090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:22.441051 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:24.416910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:24.416878 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:25.397841 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:25.397806 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:26.793123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:26.793083 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:28.660493 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:28.660454 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:30.976357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:30.976321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:31.111648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:31.111613 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:32.708401 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:32.708354 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:33.324190 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:33.324073 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:33.326864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:33.326839 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:49:33.327051 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:33.327029 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" containerID="cri-o://3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81" gracePeriod=30 Nov 03 09:49:33.840572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:33.840541 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:34.100883 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:34.100783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:35.560196 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:49:35.560134 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:49:37.067789 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:37.067766 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:49:37.118912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:37.118890 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:49:38.586807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.586783 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:49:38.688630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.688810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688644 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.688810 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688754 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.688904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688805 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.688904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688857 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.688904 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688857 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:49:38.689018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688910 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.689018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688927 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:49:38.689018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.688997 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.689202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689049 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.689202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689065 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:49:38.689202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689094 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\") pod \"1efb044f-7641-439b-97d4-ad5bd9928238\" (UID: \"1efb044f-7641-439b-97d4-ad5bd9928238\") " Nov 03 09:49:38.689361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689220 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:49:38.689361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:49:38.689607 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689590 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.689694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689612 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.689694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689625 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.689694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689638 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1efb044f-7641-439b-97d4-ad5bd9928238-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.689694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.689650 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.691271 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.691239 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "kube-api-access-tgc88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:49:38.691382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.691284 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:49:38.691629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.691609 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:49:38.691693 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.691625 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo" pod "1efb044f-7641-439b-97d4-ad5bd9928238" (UID: "1efb044f-7641-439b-97d4-ad5bd9928238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:49:38.791330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.791271 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgc88\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-kube-api-access-tgc88\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.791330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.791308 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1efb044f-7641-439b-97d4-ad5bd9928238-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.791330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.791322 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1efb044f-7641-439b-97d4-ad5bd9928238-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.791330 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.791335 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1efb044f-7641-439b-97d4-ad5bd9928238-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:49:38.973818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:38.973741 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:39.259122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.259038 4543 generic.go:356] "Generic (PLEG): container finished" podID="1efb044f-7641-439b-97d4-ad5bd9928238" containerID="3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81" exitCode=0 Nov 03 09:49:39.259122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.259090 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" Nov 03 09:49:39.259122 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.259096 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerDied","Data":"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81"} Nov 03 09:49:39.259390 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.259167 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k" event={"ID":"1efb044f-7641-439b-97d4-ad5bd9928238","Type":"ContainerDied","Data":"c5e672786411ca1dd6f5e36c2a1a0ff0c9071fd1d6040637388ca5e693ce9031"} Nov 03 09:49:39.259390 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.259190 4543 scope.go:122] "RemoveContainer" containerID="3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81" Nov 03 09:49:39.269856 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.269838 4543 scope.go:122] "RemoveContainer" containerID="f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66" Nov 03 09:49:39.278179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.278163 4543 scope.go:122] "RemoveContainer" containerID="3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81" Nov 03 09:49:39.278444 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:49:39.278426 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81\": container with ID starting with 3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81 not found: ID does not exist" containerID="3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81" Nov 03 09:49:39.278500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.278453 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81"} err="failed to get container status \"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81\": rpc error: code = NotFound desc = could not find container \"3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81\": container with ID starting with 3498bf90bf268ba274773d699323da5ea40685cd407eead43e9bb08f6259bc81 not found: ID does not exist" Nov 03 09:49:39.278500 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.278474 4543 scope.go:122] "RemoveContainer" containerID="f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66" Nov 03 09:49:39.278707 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:49:39.278687 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66\": container with ID starting with f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66 not found: ID does not exist" containerID="f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66" Nov 03 09:49:39.278772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.278717 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66"} err="failed to get container status \"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66\": rpc error: code = NotFound desc = could not find container \"f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66\": container with ID starting with f71b7d1e061c31f7cbe028f720976f6bfee24483f1b9bb84654be16ee3ba0e66 not found: ID does not exist" Nov 03 09:49:39.429194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.429153 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:49:39.479116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:39.479078 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-465707-openshift-default-7d669fd9d7-kl99k"] Nov 03 09:49:40.451416 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:40.451372 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" path="/var/lib/kubelet/pods/1efb044f-7641-439b-97d4-ad5bd9928238/volumes" Nov 03 09:49:41.098844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:41.098807 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:41.986270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:41.986236 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:42.129973 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:42.129944 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:44.393133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:44.393098 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:46.315182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:46.315132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:46.858798 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:46.858765 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:47.700941 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:47.700910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:47.887820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:47.887787 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:49.079557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:49.079525 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:49.516791 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:49.516757 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:52.134283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:52.134245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:52.276415 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:52.276374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:52.432312 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:52.432233 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:54.943645 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:54.943605 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:55.356737 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:55.356634 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:55.540248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:55.540219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:49:59.949281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:49:59.949246 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:04.958600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:04.958566 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:05.561765 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:05.561729 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:50:07.538203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:07.538170 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:08.331588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:08.331539 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:09.936065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:09.936020 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:15.047530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:15.047477 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:19.948069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:19.948031 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:24.758407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:24.758361 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:24.926099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:24.926067 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:29.883604 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:29.883560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:34.828031 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:34.827999 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:35.561358 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:35.561323 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:50:39.078885 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.078854 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:50:39.079406 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.079077 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" podUID="63c4dae5-7266-4d36-a027-d47fb5dcc939" containerName="maas-api" containerID="cri-o://4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c" gracePeriod=30 Nov 03 09:50:39.174347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.174309 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:39.368475 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.368452 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:50:39.424421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.424384 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\") pod \"63c4dae5-7266-4d36-a027-d47fb5dcc939\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " Nov 03 09:50:39.424583 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.424535 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\" (UniqueName: \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\") pod \"63c4dae5-7266-4d36-a027-d47fb5dcc939\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " Nov 03 09:50:39.424640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.424623 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") pod \"63c4dae5-7266-4d36-a027-d47fb5dcc939\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " Nov 03 09:50:39.424696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.424679 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\") pod \"63c4dae5-7266-4d36-a027-d47fb5dcc939\" (UID: \"63c4dae5-7266-4d36-a027-d47fb5dcc939\") " Nov 03 09:50:39.424753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.424733 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca" pod "63c4dae5-7266-4d36-a027-d47fb5dcc939" (UID: "63c4dae5-7266-4d36-a027-d47fb5dcc939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:50:39.425043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.425022 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4dae5-7266-4d36-a027-d47fb5dcc939-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:39.427095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.427000 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls" pod "63c4dae5-7266-4d36-a027-d47fb5dcc939" (UID: "63c4dae5-7266-4d36-a027-d47fb5dcc939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:50:39.427095 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.427020 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs" pod "63c4dae5-7266-4d36-a027-d47fb5dcc939" (UID: "63c4dae5-7266-4d36-a027-d47fb5dcc939"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:50:39.427278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.427191 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt" pod "63c4dae5-7266-4d36-a027-d47fb5dcc939" (UID: "63c4dae5-7266-4d36-a027-d47fb5dcc939"). InnerVolumeSpecName "kube-api-access-ft9jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:50:39.526695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.526660 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9jt\" (UniqueName: \"kubernetes.io/projected/63c4dae5-7266-4d36-a027-d47fb5dcc939-kube-api-access-ft9jt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:39.526695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.526694 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:39.526695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.526703 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c4dae5-7266-4d36-a027-d47fb5dcc939-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:39.634179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.634114 4543 generic.go:356] "Generic (PLEG): container finished" podID="63c4dae5-7266-4d36-a027-d47fb5dcc939" containerID="4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c" exitCode=0 Nov 03 09:50:39.634383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.634199 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" Nov 03 09:50:39.634383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.634201 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" event={"ID":"63c4dae5-7266-4d36-a027-d47fb5dcc939","Type":"ContainerDied","Data":"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c"} Nov 03 09:50:39.634383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.634265 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw" event={"ID":"63c4dae5-7266-4d36-a027-d47fb5dcc939","Type":"ContainerDied","Data":"48376281800afdcb1d3b50cf077dd53d411a7c5bb806d3cf6ea826dec7e35101"} Nov 03 09:50:39.634383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.634293 4543 scope.go:122] "RemoveContainer" containerID="4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c" Nov 03 09:50:39.649681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.649656 4543 scope.go:122] "RemoveContainer" containerID="4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c" Nov 03 09:50:39.651009 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:39.650092 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c\": container with ID starting with 4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c not found: ID does not exist" containerID="4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c" Nov 03 09:50:39.651009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.650131 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c"} err="failed to get container status \"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c\": rpc error: code = NotFound desc = could not find container \"4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c\": container with ID starting with 4bfaa261632926f2cd9a26d9993b935bc4f8d2face9eb8a653619f82b092998c not found: ID does not exist" Nov 03 09:50:39.803463 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.803430 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:50:39.810868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:39.810838 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-3df1cc-5b76bc4b79-wdkcw"] Nov 03 09:50:40.451371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:40.451338 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c4dae5-7266-4d36-a027-d47fb5dcc939" path="/var/lib/kubelet/pods/63c4dae5-7266-4d36-a027-d47fb5dcc939/volumes" Nov 03 09:50:40.956691 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:40.956643 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:42.816721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:42.816691 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:42.952813 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:42.952779 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:44.445052 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:44.445016 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:44.977833 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:44.977803 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:45.407901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:45.407863 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:46.235424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:46.235391 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:50.502602 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:50.502517 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:51.423173 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:51.423097 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:51.821426 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:51.821347 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:50:51.821814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:51.821544 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerName="discovery" containerID="cri-o://499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0" gracePeriod=30 Nov 03 09:50:52.093996 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.093916 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:50:52.175976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.175933 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176015 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176255 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176320 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176310 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176355 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176440 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176427 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\") pod \"1faebc12-103c-445c-9f2f-449ecb1e23cf\" (UID: \"1faebc12-103c-445c-9f2f-449ecb1e23cf\") " Nov 03 09:50:52.176505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176458 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:50:52.177090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.176947 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-ca-configmap\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.178461 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.178437 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:50:52.178763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.178736 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:50:52.178920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.178905 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:50:52.179059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.179039 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:50:52.179104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.179044 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:50:52.179236 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.179220 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws" pod "1faebc12-103c-445c-9f2f-449ecb1e23cf" (UID: "1faebc12-103c-445c-9f2f-449ecb1e23cf"). InnerVolumeSpecName "kube-api-access-m2cws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:50:52.278002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.277969 4543 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faebc12-103c-445c-9f2f-449ecb1e23cf-local-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.278002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.277996 4543 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-cacerts\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.278002 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.278005 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.278265 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.278015 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2cws\" (UniqueName: \"kubernetes.io/projected/1faebc12-103c-445c-9f2f-449ecb1e23cf-kube-api-access-m2cws\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.278265 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.278026 4543 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-csr-dns-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.278265 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.278035 4543 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1faebc12-103c-445c-9f2f-449ecb1e23cf-istio-kubeconfig\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:50:52.713516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.713429 4543 generic.go:356] "Generic (PLEG): container finished" podID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerID="499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0" exitCode=0 Nov 03 09:50:52.713516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.713473 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" Nov 03 09:50:52.713516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.713509 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" event={"ID":"1faebc12-103c-445c-9f2f-449ecb1e23cf","Type":"ContainerDied","Data":"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0"} Nov 03 09:50:52.713779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.713554 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg" event={"ID":"1faebc12-103c-445c-9f2f-449ecb1e23cf","Type":"ContainerDied","Data":"7f948f830028551035a12aec79917158829d2fd0b2668dc592974dd409ec0e1a"} Nov 03 09:50:52.713779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.713575 4543 scope.go:122] "RemoveContainer" containerID="499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0" Nov 03 09:50:52.723994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.723974 4543 scope.go:122] "RemoveContainer" containerID="499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0" Nov 03 09:50:52.724341 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:52.724314 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0\": container with ID starting with 499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0 not found: ID does not exist" containerID="499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0" Nov 03 09:50:52.724411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.724352 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0"} err="failed to get container status \"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0\": rpc error: code = NotFound desc = could not find container \"499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0\": container with ID starting with 499e2f5537e749dd035e01c754a4e79876afbbe8f7ab0404be19a4d035f21ab0 not found: ID does not exist" Nov 03 09:50:52.886272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.886229 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:50:52.936640 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:52.936605 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-knkzg"] Nov 03 09:50:54.451664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:54.451626 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" path="/var/lib/kubelet/pods/1faebc12-103c-445c-9f2f-449ecb1e23cf/volumes" Nov 03 09:50:55.470409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.470366 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:50:55.471613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471586 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:50:55.471613 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471612 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471646 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471654 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c149402-cbec-4cd9-ae0f-9f13b55717d0" containerName="cleanup" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471674 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471681 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471692 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471699 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471724 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471731 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471765 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471772 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.471802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471807 4543 cpu_manager.go:414] "containerMap: removing container" podUID="63c4dae5-7266-4d36-a027-d47fb5dcc939" containerName="maas-api" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471814 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c4dae5-7266-4d36-a027-d47fb5dcc939" containerName="maas-api" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471862 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerName="discovery" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.471869 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerName="discovery" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472102 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c4dae5-7266-4d36-a027-d47fb5dcc939" containerName="maas-api" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472130 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="87449f85-afd4-47a1-ad8d-bee41013ba65" containerName="istio-proxy" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472166 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faebc12-103c-445c-9f2f-449ecb1e23cf" containerName="discovery" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472177 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ffa15-de72-4ca1-a28b-ed9c19c7ee31" containerName="cleanup" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472193 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472202 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df46ae1-d457-4d77-a6ae-c5c7949309e4" containerName="istio-proxy" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472219 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.472364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.472229 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efb044f-7641-439b-97d4-ad5bd9928238" containerName="istio-proxy" Nov 03 09:50:55.478646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.478623 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.529177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.529129 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:50:55.554938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.554905 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:55.576720 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.576692 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"kube-root-ca.crt\"" Nov 03 09:50:55.576720 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.576697 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"default-dockercfg-5g5dj\"" Nov 03 09:50:55.582701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.582683 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"rate-test-model-9ed110-kserve-self-signed-certs\"" Nov 03 09:50:55.582874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.582859 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"openshift-service-ca.crt\"" Nov 03 09:50:55.617244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617204 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.617506 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617261 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.617506 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617321 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.617506 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617441 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcs8j\" (UniqueName: \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.617778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617537 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.617778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.617638 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719263 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcs8j\" (UniqueName: \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719365 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719457 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719500 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719531 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719591 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719819 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719887 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719847 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.719967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.719943 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.722090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.722040 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.722194 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.722132 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.783552 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.783518 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcs8j\" (UniqueName: \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\") pod \"rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:55.792624 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:55.792566 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:56.171326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.171297 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:50:56.172365 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:50:56.172338 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bc4de3c_eae9_4d42_8252_77295e318769.slice/crio-ec456131cb82a6cdad3373b6b15be2b99fae9cd298a4b0203a38940ab18153a6 WatchSource:0}: Error finding container ec456131cb82a6cdad3373b6b15be2b99fae9cd298a4b0203a38940ab18153a6: Status 404 returned error can't find the container with id ec456131cb82a6cdad3373b6b15be2b99fae9cd298a4b0203a38940ab18153a6 Nov 03 09:50:56.250882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.250851 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:50:56.254995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.254972 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.309443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.309416 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f8439f5b-serving-cert\"" Nov 03 09:50:56.312397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.312370 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:50:56.426860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.426776 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.427038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.426904 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88cgq\" (UniqueName: \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.427038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.426999 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.427154 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.427085 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.441424 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.441393 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:56.528577 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.528543 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88cgq\" (UniqueName: \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.529103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.528598 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.529103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.528643 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.529103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.528748 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.529103 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:56.528877 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-serving-cert: secret "maas-api-e2e-mt-f8439f5b-serving-cert" not found Nov 03 09:50:56.529103 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:50:56.528945 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls podName:185a3ba7-a97f-418a-b243-9647ffe018ed nodeName:}" failed. No retries permitted until 2026-11-03 09:50:57.028925276 +0000 UTC m=+4280.696965993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls") pod "maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" (UID: "185a3ba7-a97f-418a-b243-9647ffe018ed") : secret "maas-api-e2e-mt-f8439f5b-serving-cert" not found Nov 03 09:50:56.529629 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.529609 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.531397 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.531375 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.587750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.587708 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88cgq\" (UniqueName: \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:56.602201 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.602133 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:56.728823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.728738 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:56.738348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.738315 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" event={"ID":"3bc4de3c-eae9-4d42-8252-77295e318769","Type":"ContainerStarted","Data":"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466"} Nov 03 09:50:56.738494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.738362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" event={"ID":"3bc4de3c-eae9-4d42-8252-77295e318769","Type":"ContainerStarted","Data":"ec456131cb82a6cdad3373b6b15be2b99fae9cd298a4b0203a38940ab18153a6"} Nov 03 09:50:56.793820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.793785 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:50:56.857553 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.857493 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" podStartSLOduration=1.857480013 podStartE2EDuration="1.857480013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:50:56.856474406 +0000 UTC m=+4280.524515147" watchObservedRunningTime="2026-11-03 09:50:56.857480013 +0000 UTC m=+4280.525520751" Nov 03 09:50:56.892088 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:56.892051 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:50:57.035009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.034924 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:57.037562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.037535 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") pod \"maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:57.169794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.169751 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:57.598695 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.598666 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:50:57.600155 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:50:57.600117 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185a3ba7_a97f_418a_b243_9647ffe018ed.slice/crio-b259f1ce795a1118423be3cb3f792bf8b54de2dd8436d5f15437c374ae272958 WatchSource:0}: Error finding container b259f1ce795a1118423be3cb3f792bf8b54de2dd8436d5f15437c374ae272958: Status 404 returned error can't find the container with id b259f1ce795a1118423be3cb3f792bf8b54de2dd8436d5f15437c374ae272958 Nov 03 09:50:57.601647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.601627 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:50:57.747131 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:57.747089 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerStarted","Data":"b259f1ce795a1118423be3cb3f792bf8b54de2dd8436d5f15437c374ae272958"} Nov 03 09:50:58.754722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:58.754617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerStarted","Data":"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f"} Nov 03 09:50:58.755526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:58.754837 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:50:58.872930 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:50:58.872877 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" podStartSLOduration=1.9740681150000001 podStartE2EDuration="2.872863007s" totalImagesPullingTime="898.794892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:50:58.872036904 +0000 UTC m=+4282.540077646" watchObservedRunningTime="2026-11-03 09:50:58.872863007 +0000 UTC m=+4282.540903783" Nov 03 09:51:01.575720 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:01.575688 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:04.474772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:04.474742 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:04.762615 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:04.762523 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.141:8443/health\": dial tcp 10.134.0.141:8443: connect: connection refused" Nov 03 09:51:05.290672 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:05.290638 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:05.420893 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:05.420853 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:05.560284 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:51:05.560200 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:51:05.793646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:05.793617 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:51:05.808982 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:05.808951 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:51:06.805525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:06.805491 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:51:06.985007 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:06.984976 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:07.887496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:07.887444 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:08.697978 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:08.697947 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:08.819192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:08.819133 4543 generic.go:356] "Generic (PLEG): container finished" podID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerID="47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f" exitCode=1 Nov 03 09:51:08.819192 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:08.819174 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerDied","Data":"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f"} Nov 03 09:51:08.819587 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:08.819572 4543 scope.go:122] "RemoveContainer" containerID="47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f" Nov 03 09:51:09.826813 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:09.826782 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerStarted","Data":"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89"} Nov 03 09:51:09.827247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:09.827024 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:51:10.911836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:10.911803 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:12.334258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:12.334222 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:15.837091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:15.837061 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:51:17.342035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:17.342002 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:17.472182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:17.472126 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:22.513457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:22.513420 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:25.107414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:25.107384 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:27.305901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:27.305861 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:27.436448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:27.436409 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:27.569359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:27.569269 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:28.431387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:28.431354 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:28.564857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:28.564822 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:28.841966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:28.841872 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:32.544903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:32.544865 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:35.397494 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:35.397459 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:35.533027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:35.532992 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:35.561546 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:51:35.561506 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:51:37.635372 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:37.635345 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:42.653991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:42.653955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:47.743157 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:47.743100 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:52.732578 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:52.732543 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:51:57.911532 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:51:57.911499 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:02.854571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:02.854537 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:04.669866 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:04.669827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:05.478297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:05.478268 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:05.562878 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:52:05.562842 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:52:05.614274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:05.614239 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:06.068573 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:06.068527 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:07.421905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.421871 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 09:52:07.422413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.422082 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" podUID="14c04f07-2576-478b-a56a-c12c3e322119" containerName="maas-api" containerID="cri-o://10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4" gracePeriod=30 Nov 03 09:52:07.510291 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.510259 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:07.710658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.710630 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 09:52:07.812917 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.812877 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\") pod \"14c04f07-2576-478b-a56a-c12c3e322119\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " Nov 03 09:52:07.813133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.812962 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\" (UniqueName: \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\") pod \"14c04f07-2576-478b-a56a-c12c3e322119\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " Nov 03 09:52:07.813133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.813017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") pod \"14c04f07-2576-478b-a56a-c12c3e322119\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " Nov 03 09:52:07.813133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.813112 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\") pod \"14c04f07-2576-478b-a56a-c12c3e322119\" (UID: \"14c04f07-2576-478b-a56a-c12c3e322119\") " Nov 03 09:52:07.813999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.813952 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca" pod "14c04f07-2576-478b-a56a-c12c3e322119" (UID: "14c04f07-2576-478b-a56a-c12c3e322119"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:52:07.817058 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.817027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls" pod "14c04f07-2576-478b-a56a-c12c3e322119" (UID: "14c04f07-2576-478b-a56a-c12c3e322119"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:52:07.817919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.817888 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p" pod "14c04f07-2576-478b-a56a-c12c3e322119" (UID: "14c04f07-2576-478b-a56a-c12c3e322119"). InnerVolumeSpecName "kube-api-access-4962p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:52:07.825357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.825286 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs" pod "14c04f07-2576-478b-a56a-c12c3e322119" (UID: "14c04f07-2576-478b-a56a-c12c3e322119"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:52:07.914544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.914512 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4962p\" (UniqueName: \"kubernetes.io/projected/14c04f07-2576-478b-a56a-c12c3e322119-kube-api-access-4962p\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:07.914544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.914541 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:07.914544 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.914550 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14c04f07-2576-478b-a56a-c12c3e322119-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:07.914787 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:07.914560 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14c04f07-2576-478b-a56a-c12c3e322119-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:08.211895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.211849 4543 generic.go:356] "Generic (PLEG): container finished" podID="14c04f07-2576-478b-a56a-c12c3e322119" containerID="10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4" exitCode=0 Nov 03 09:52:08.212074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.211900 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" Nov 03 09:52:08.212074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.211914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" event={"ID":"14c04f07-2576-478b-a56a-c12c3e322119","Type":"ContainerDied","Data":"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4"} Nov 03 09:52:08.212074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.211978 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh" event={"ID":"14c04f07-2576-478b-a56a-c12c3e322119","Type":"ContainerDied","Data":"a4dc12c489819451005baa415c57935411dcb36f8a02957b27a7778213420637"} Nov 03 09:52:08.212074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.212004 4543 scope.go:122] "RemoveContainer" containerID="10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4" Nov 03 09:52:08.224408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.224390 4543 scope.go:122] "RemoveContainer" containerID="10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4" Nov 03 09:52:08.224706 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:52:08.224686 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4\": container with ID starting with 10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4 not found: ID does not exist" containerID="10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4" Nov 03 09:52:08.224780 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.224715 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4"} err="failed to get container status \"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4\": rpc error: code = NotFound desc = could not find container \"10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4\": container with ID starting with 10bf10cd3bc2fde012f5831ba9a31f07fbc5c4131800c4a814623c18d79bfde4 not found: ID does not exist" Nov 03 09:52:08.239992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.239961 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:08.427733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.427699 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 09:52:08.436558 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.436522 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-26e861-fd8d5c99f-vfqsh"] Nov 03 09:52:08.452997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:08.452968 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c04f07-2576-478b-a56a-c12c3e322119" path="/var/lib/kubelet/pods/14c04f07-2576-478b-a56a-c12c3e322119/volumes" Nov 03 09:52:09.329880 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:09.329844 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:12.637979 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:12.637946 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:12.809316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:12.809240 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:12.944443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:12.944357 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:14.376886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:14.376851 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:14.511913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:14.511882 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:16.106561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:16.106533 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:19.632099 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:19.632005 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:24.634984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:24.634951 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:25.463905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:25.463853 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:26.728006 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:26.727967 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:26.730069 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:26.730043 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:52:26.730316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:26.730293 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" containerID="cri-o://c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e" gracePeriod=30 Nov 03 09:52:29.632843 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:29.632802 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:32.000439 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.000383 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:52:32.110497 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110464 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110509 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110536 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110688 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110716 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110747 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.110794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110765 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.110794 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110764 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.111041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110829 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.111041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110880 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\") pod \"deb60d98-08a4-48da-9642-0d3abed943c8\" (UID: \"deb60d98-08a4-48da-9642-0d3abed943c8\") " Nov 03 09:52:32.111041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.110976 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.111239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111053 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:52:32.111239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111133 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.111571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111394 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb60d98-08a4-48da-9642-0d3abed943c8-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.111571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111578 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.111571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111594 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.111571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111609 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.111571 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.111622 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.113039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.113009 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:52:32.113301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.113277 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:52:32.113581 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.113564 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:52:32.113638 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.113585 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx" pod "deb60d98-08a4-48da-9642-0d3abed943c8" (UID: "deb60d98-08a4-48da-9642-0d3abed943c8"). InnerVolumeSpecName "kube-api-access-zrdnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:52:32.213267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.213236 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb60d98-08a4-48da-9642-0d3abed943c8-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.213267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.213264 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.213267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.213273 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb60d98-08a4-48da-9642-0d3abed943c8-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.213267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.213283 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdnx\" (UniqueName: \"kubernetes.io/projected/deb60d98-08a4-48da-9642-0d3abed943c8-kube-api-access-zrdnx\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:52:32.361676 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.361643 4543 generic.go:356] "Generic (PLEG): container finished" podID="deb60d98-08a4-48da-9642-0d3abed943c8" containerID="c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e" exitCode=0 Nov 03 09:52:32.361867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.361694 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" Nov 03 09:52:32.361867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.361730 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" event={"ID":"deb60d98-08a4-48da-9642-0d3abed943c8","Type":"ContainerDied","Data":"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e"} Nov 03 09:52:32.361867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.361789 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5" event={"ID":"deb60d98-08a4-48da-9642-0d3abed943c8","Type":"ContainerDied","Data":"32eece940b1043f2908ce7a106d9b10db64a5c1dd88ccd0aef9ef76825166fe3"} Nov 03 09:52:32.361867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.361818 4543 scope.go:122] "RemoveContainer" containerID="c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e" Nov 03 09:52:32.372380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.372361 4543 scope.go:122] "RemoveContainer" containerID="c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e" Nov 03 09:52:32.372655 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:52:32.372629 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e\": container with ID starting with c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e not found: ID does not exist" containerID="c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e" Nov 03 09:52:32.372699 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.372665 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e"} err="failed to get container status \"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e\": rpc error: code = NotFound desc = could not find container \"c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e\": container with ID starting with c7fa5facef8694641fa32facadc31f4b6d729a44e664342dd6b32c02028d680e not found: ID does not exist" Nov 03 09:52:32.533179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.533130 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:52:32.537659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:32.537627 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-l48g5"] Nov 03 09:52:34.451082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:34.451044 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" path="/var/lib/kubelet/pods/deb60d98-08a4-48da-9642-0d3abed943c8/volumes" Nov 03 09:52:34.637643 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:34.637597 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:35.560936 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:52:35.560903 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:52:39.659600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:39.659556 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:44.640127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:44.640093 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:49.563674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:49.563633 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:54.406336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:54.406291 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:54.548862 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:54.548831 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:52:59.545370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:52:59.545335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:04.543724 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:04.543687 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:05.561075 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:05.561041 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:53:08.609540 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:08.609499 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:09.362465 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:09.362430 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:09.543030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:09.542995 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:13.628976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.628907 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:53:13.629972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.629872 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" containerID="cri-o://5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392" gracePeriod=30 Nov 03 09:53:13.687980 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.687940 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 09:53:13.688199 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.688023 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:53:13.688267 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.688237 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" containerID="cri-o://143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d" gracePeriod=30 Nov 03 09:53:13.688599 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.688574 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" containerID="cri-o://62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d" gracePeriod=30 Nov 03 09:53:13.733229 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.733200 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:53:13.733430 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:13.733408 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" containerID="cri-o://cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab" gracePeriod=30 Nov 03 09:53:14.658092 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:14.658059 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:15.147999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.147955 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-65rgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:53:15.148208 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.148026 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:53:15.528046 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.527969 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:53:15.528046 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.528035 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:53:15.793090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.792996 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d56dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:53:15.793090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.793053 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:53:15.828111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.828080 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7hddr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:53:15.828300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:15.828132 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:53:18.589485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:18.589440 4543 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal" podUID="20fdfbfab5aa17ccf7324ae585a08734" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 09:53:19.030537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.030509 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:53:19.056318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.056286 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 09:53:19.060545 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.060520 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:53:19.171819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.171785 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.171835 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.171864 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.171914 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.171957 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172030 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172001 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172032 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172058 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172104 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172179 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172225 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172254 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172289 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172357 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172320 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172406 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172452 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172485 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172522 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172576 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172643 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172683 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172731 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.172777 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172761 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.173258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172811 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\") pod \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\" (UID: \"cd0b110a-bc54-4a4b-a48e-11e2383a616d\") " Nov 03 09:53:19.173258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172857 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.173258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172886 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\") pod \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\" (UID: \"19fd6e4e-8a01-4955-b8a8-a0c907791f05\") " Nov 03 09:53:19.173258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.172928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\") pod \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\" (UID: \"63cfbcbd-bdac-4f3a-94a7-44edd0230829\") " Nov 03 09:53:19.174387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.173840 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.174387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.173844 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.174387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.174447 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.174387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.174871 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175090 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175108 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:53:19.175442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175257 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:53:19.175442 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175426 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:53:19.175752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175673 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175689 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175842 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.175928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.175885 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.176043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.176006 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.176043 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.176027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.176248 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.176228 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.177047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.176801 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "kube-api-access-rdfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.177047 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.176886 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.179079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.179050 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "kube-api-access-wtsv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.179212 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.179187 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.180352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180326 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:53:19.180508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180482 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.180694 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180659 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.180802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180729 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m" pod "19fd6e4e-8a01-4955-b8a8-a0c907791f05" (UID: "19fd6e4e-8a01-4955-b8a8-a0c907791f05"). InnerVolumeSpecName "kube-api-access-8j59m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.180802 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180781 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:53:19.181021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.180993 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo" pod "63cfbcbd-bdac-4f3a-94a7-44edd0230829" (UID: "63cfbcbd-bdac-4f3a-94a7-44edd0230829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:53:19.181021 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.181065 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token" pod "cd0b110a-bc54-4a4b-a48e-11e2383a616d" (UID: "cd0b110a-bc54-4a4b-a48e-11e2383a616d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.274752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274717 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.274752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274749 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.274752 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274758 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274769 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274778 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274786 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274796 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274805 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274819 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274828 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274836 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274844 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274853 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsv7\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-kube-api-access-wtsv7\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274863 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j59m\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-kube-api-access-8j59m\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274871 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274879 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274888 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274895 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274903 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274911 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd0b110a-bc54-4a4b-a48e-11e2383a616d-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274918 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274926 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274934 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274942 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdfjr\" (UniqueName: \"kubernetes.io/projected/cd0b110a-bc54-4a4b-a48e-11e2383a616d-kube-api-access-rdfjr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274950 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cfbcbd-bdac-4f3a-94a7-44edd0230829-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274958 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.275061 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.274967 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fd6e4e-8a01-4955-b8a8-a0c907791f05-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.342832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.342804 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:53:19.477107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477006 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477107 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477193 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477221 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477263 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477302 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477450 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477476 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\") pod \"3c4e3da3-f75a-448b-87eb-281f0d724806\" (UID: \"3c4e3da3-f75a-448b-87eb-281f0d724806\") " Nov 03 09:53:19.477614 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477545 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.477828 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477729 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:53:19.477828 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.477828 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477746 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.478000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477942 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.478000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477964 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.478000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477977 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e3da3-f75a-448b-87eb-281f0d724806-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.478000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.477990 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.478247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.478022 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.480206 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.479985 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.480327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.480309 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "kube-api-access-4v9cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:19.480410 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.480365 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:53:19.480454 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.480413 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo" pod "3c4e3da3-f75a-448b-87eb-281f0d724806" (UID: "3c4e3da3-f75a-448b-87eb-281f0d724806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:53:19.578936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.578899 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9cd\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-kube-api-access-4v9cd\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.578936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.578940 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.579217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.578957 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.579217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.578972 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e3da3-f75a-448b-87eb-281f0d724806-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.579217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.578987 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e3da3-f75a-448b-87eb-281f0d724806-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:19.631840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.631801 4543 generic.go:356] "Generic (PLEG): container finished" podID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerID="62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d" exitCode=0 Nov 03 09:53:19.632362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.631850 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" Nov 03 09:53:19.632362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.631874 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" event={"ID":"cd0b110a-bc54-4a4b-a48e-11e2383a616d","Type":"ContainerDied","Data":"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d"} Nov 03 09:53:19.632362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.631936 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg" event={"ID":"cd0b110a-bc54-4a4b-a48e-11e2383a616d","Type":"ContainerDied","Data":"015092459a167799d7a6f6d5389d7efab0f3ac85bc3b50602ae1a3d9efbd3099"} Nov 03 09:53:19.632362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.631974 4543 scope.go:122] "RemoveContainer" containerID="62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d" Nov 03 09:53:19.633660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.633635 4543 generic.go:356] "Generic (PLEG): container finished" podID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerID="143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d" exitCode=0 Nov 03 09:53:19.633811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.633678 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" Nov 03 09:53:19.633811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.633740 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" event={"ID":"63cfbcbd-bdac-4f3a-94a7-44edd0230829","Type":"ContainerDied","Data":"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d"} Nov 03 09:53:19.633811 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.633793 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z" event={"ID":"63cfbcbd-bdac-4f3a-94a7-44edd0230829","Type":"ContainerDied","Data":"f38bc6597d4120f9d5fbd632bceafd89fb54f67d80c978d4c81d411d78e72f54"} Nov 03 09:53:19.638882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.638855 4543 generic.go:356] "Generic (PLEG): container finished" podID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerID="cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab" exitCode=0 Nov 03 09:53:19.639029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.638903 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" Nov 03 09:53:19.639029 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.638977 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" event={"ID":"19fd6e4e-8a01-4955-b8a8-a0c907791f05","Type":"ContainerDied","Data":"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab"} Nov 03 09:53:19.639196 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.639029 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn" event={"ID":"19fd6e4e-8a01-4955-b8a8-a0c907791f05","Type":"ContainerDied","Data":"b65f1b24eabfe81112e51960adf2c10e4af7fa4050d88e828c199017666b4371"} Nov 03 09:53:19.647799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.647769 4543 generic.go:356] "Generic (PLEG): container finished" podID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerID="5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392" exitCode=0 Nov 03 09:53:19.647965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.647809 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" event={"ID":"3c4e3da3-f75a-448b-87eb-281f0d724806","Type":"ContainerDied","Data":"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392"} Nov 03 09:53:19.647965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.647844 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" Nov 03 09:53:19.647965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.647859 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr" event={"ID":"3c4e3da3-f75a-448b-87eb-281f0d724806","Type":"ContainerDied","Data":"3dcb0db1b691999cf114647f3881cc9517d9522acedd996d4a74bb77902247e4"} Nov 03 09:53:19.650224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.650207 4543 scope.go:122] "RemoveContainer" containerID="62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d" Nov 03 09:53:19.650607 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:19.650579 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d\": container with ID starting with 62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d not found: ID does not exist" containerID="62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d" Nov 03 09:53:19.650734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.650619 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d"} err="failed to get container status \"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d\": rpc error: code = NotFound desc = could not find container \"62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d\": container with ID starting with 62a2a44a7abd7dfc4fef45348a14292b4aea8a564b072d11891b972ed3c8547d not found: ID does not exist" Nov 03 09:53:19.650734 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.650645 4543 scope.go:122] "RemoveContainer" containerID="143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d" Nov 03 09:53:19.656517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.656489 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:19.662659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.662636 4543 scope.go:122] "RemoveContainer" containerID="143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d" Nov 03 09:53:19.662967 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:19.662946 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d\": container with ID starting with 143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d not found: ID does not exist" containerID="143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d" Nov 03 09:53:19.663037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.662980 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d"} err="failed to get container status \"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d\": rpc error: code = NotFound desc = could not find container \"143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d\": container with ID starting with 143987b861cf7530261dd299f73ffe86638f6e56b907655d9efa607b4c45193d not found: ID does not exist" Nov 03 09:53:19.663037 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.663005 4543 scope.go:122] "RemoveContainer" containerID="cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab" Nov 03 09:53:19.672359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.672338 4543 scope.go:122] "RemoveContainer" containerID="cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab" Nov 03 09:53:19.672679 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:19.672647 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab\": container with ID starting with cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab not found: ID does not exist" containerID="cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab" Nov 03 09:53:19.672761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.672690 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab"} err="failed to get container status \"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab\": rpc error: code = NotFound desc = could not find container \"cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab\": container with ID starting with cbff50cb02de1b39483c7aabed8d254968ff3c8be3e0fdf998c982e34bbf16ab not found: ID does not exist" Nov 03 09:53:19.672761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.672715 4543 scope.go:122] "RemoveContainer" containerID="5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392" Nov 03 09:53:19.682772 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.682746 4543 scope.go:122] "RemoveContainer" containerID="5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392" Nov 03 09:53:19.683078 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:19.683056 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392\": container with ID starting with 5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392 not found: ID does not exist" containerID="5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392" Nov 03 09:53:19.683179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.683092 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392"} err="failed to get container status \"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392\": rpc error: code = NotFound desc = could not find container \"5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392\": container with ID starting with 5fcd677cb0b1f313b86e865f3c42c7887a1866d82eef67cd8d1b308fbdae2392 not found: ID does not exist" Nov 03 09:53:19.809055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.808971 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 09:53:19.815315 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.815284 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-65rgg"] Nov 03 09:53:19.927935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.927895 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:53:19.934929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:19.934899 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-hxf4z"] Nov 03 09:53:20.048412 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.048383 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:53:20.054951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.054922 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7hddr"] Nov 03 09:53:20.166380 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.166348 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:53:20.173759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.173729 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d56dn"] Nov 03 09:53:20.451660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.451566 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" path="/var/lib/kubelet/pods/19fd6e4e-8a01-4955-b8a8-a0c907791f05/volumes" Nov 03 09:53:20.452130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.452108 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" path="/var/lib/kubelet/pods/3c4e3da3-f75a-448b-87eb-281f0d724806/volumes" Nov 03 09:53:20.452616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.452596 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" path="/var/lib/kubelet/pods/63cfbcbd-bdac-4f3a-94a7-44edd0230829/volumes" Nov 03 09:53:20.453419 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:20.453399 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" path="/var/lib/kubelet/pods/cd0b110a-bc54-4a4b-a48e-11e2383a616d/volumes" Nov 03 09:53:24.644754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:24.644717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:25.824882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:25.824846 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:29.637790 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:29.637745 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:31.870347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:31.870308 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:32.046991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:32.046961 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:34.633447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:34.633403 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:35.561002 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:35.560960 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:53:36.765384 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:36.765330 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:39.578010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:39.577979 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:43.298623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:43.298585 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:44.605183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:44.605125 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:44.735719 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:44.735690 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:44.868786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:44.868755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:49.931457 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:49.931374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:50.937433 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:50.937393 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:54.941859 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:54.941819 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:56.102370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:56.102337 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:59.256753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.256719 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:53:59.257217 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.256948 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" containerID="cri-o://226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89" gracePeriod=30 Nov 03 09:53:59.349680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.349638 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:53:59.520348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.520290 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:53:59.615348 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615301 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\") pod \"185a3ba7-a97f-418a-b243-9647ffe018ed\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " Nov 03 09:53:59.615535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615445 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\" (UniqueName: \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\") pod \"185a3ba7-a97f-418a-b243-9647ffe018ed\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " Nov 03 09:53:59.615535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615514 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") pod \"185a3ba7-a97f-418a-b243-9647ffe018ed\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " Nov 03 09:53:59.615660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615541 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\") pod \"185a3ba7-a97f-418a-b243-9647ffe018ed\" (UID: \"185a3ba7-a97f-418a-b243-9647ffe018ed\") " Nov 03 09:53:59.615717 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615677 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca" pod "185a3ba7-a97f-418a-b243-9647ffe018ed" (UID: "185a3ba7-a97f-418a-b243-9647ffe018ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:53:59.615891 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.615869 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/185a3ba7-a97f-418a-b243-9647ffe018ed-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:59.617902 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.617874 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls" pod "185a3ba7-a97f-418a-b243-9647ffe018ed" (UID: "185a3ba7-a97f-418a-b243-9647ffe018ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:53:59.618016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.617925 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs" pod "185a3ba7-a97f-418a-b243-9647ffe018ed" (UID: "185a3ba7-a97f-418a-b243-9647ffe018ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:53:59.618016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.617951 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq" pod "185a3ba7-a97f-418a-b243-9647ffe018ed" (UID: "185a3ba7-a97f-418a-b243-9647ffe018ed"). InnerVolumeSpecName "kube-api-access-88cgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:53:59.716619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.716583 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:59.716619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.716614 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/185a3ba7-a97f-418a-b243-9647ffe018ed-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:59.716619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.716624 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88cgq\" (UniqueName: \"kubernetes.io/projected/185a3ba7-a97f-418a-b243-9647ffe018ed-kube-api-access-88cgq\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:53:59.895826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.895790 4543 generic.go:356] "Generic (PLEG): container finished" podID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerID="226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89" exitCode=0 Nov 03 09:53:59.896011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.895840 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" Nov 03 09:53:59.896011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.895881 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerDied","Data":"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89"} Nov 03 09:53:59.896011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.895937 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf" event={"ID":"185a3ba7-a97f-418a-b243-9647ffe018ed","Type":"ContainerDied","Data":"b259f1ce795a1118423be3cb3f792bf8b54de2dd8436d5f15437c374ae272958"} Nov 03 09:53:59.896011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.895966 4543 scope.go:122] "RemoveContainer" containerID="226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89" Nov 03 09:53:59.905927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.905909 4543 scope.go:122] "RemoveContainer" containerID="47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f" Nov 03 09:53:59.914512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.914493 4543 scope.go:122] "RemoveContainer" containerID="226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89" Nov 03 09:53:59.914737 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:59.914719 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89\": container with ID starting with 226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89 not found: ID does not exist" containerID="226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89" Nov 03 09:53:59.914806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.914744 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89"} err="failed to get container status \"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89\": rpc error: code = NotFound desc = could not find container \"226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89\": container with ID starting with 226f210846f95392cac493eb100a327bc25211fdbad044c82daed29480cd5e89 not found: ID does not exist" Nov 03 09:53:59.914806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.914761 4543 scope.go:122] "RemoveContainer" containerID="47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f" Nov 03 09:53:59.915002 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:53:59.914986 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f\": container with ID starting with 47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f not found: ID does not exist" containerID="47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f" Nov 03 09:53:59.915045 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:53:59.915008 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f"} err="failed to get container status \"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f\": rpc error: code = NotFound desc = could not find container \"47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f\": container with ID starting with 47145694b5083fa843c07aae00bcffecd613dfb32286e227a353c5ab169e7a9f not found: ID does not exist" Nov 03 09:54:00.067171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:00.067128 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:54:00.117359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:00.117324 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f8439f5b-5c79c8bcbd-svwjf"] Nov 03 09:54:00.451580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:00.451547 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" path="/var/lib/kubelet/pods/185a3ba7-a97f-418a-b243-9647ffe018ed/volumes" Nov 03 09:54:01.262909 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:01.262859 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:05.561735 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:05.561690 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:54:06.130183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:06.130095 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:07.879104 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:07.879061 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:08.079827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:08.079772 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:09.510501 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:09.510459 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:09.759001 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:09.758967 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:10.111618 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:10.111588 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:10.534195 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:10.534092 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:11.527258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:11.527222 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:14.605860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.605825 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-41494e"] Nov 03 09:54:14.606343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606328 4543 cpu_manager.go:414] "containerMap: removing container" podUID="14c04f07-2576-478b-a56a-c12c3e322119" containerName="maas-api" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606344 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c04f07-2576-478b-a56a-c12c3e322119" containerName="maas-api" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606366 4543 cpu_manager.go:414] "containerMap: removing container" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606372 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606379 4543 cpu_manager.go:414] "containerMap: removing container" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606385 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" Nov 03 09:54:14.606393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606395 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606400 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606407 4543 cpu_manager.go:414] "containerMap: removing container" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606412 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606424 4543 cpu_manager.go:414] "containerMap: removing container" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606430 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606444 4543 cpu_manager.go:414] "containerMap: removing container" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606448 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606455 4543 cpu_manager.go:414] "containerMap: removing container" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606459 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606528 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb60d98-08a4-48da-9642-0d3abed943c8" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606537 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cfbcbd-bdac-4f3a-94a7-44edd0230829" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606542 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0b110a-bc54-4a4b-a48e-11e2383a616d" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606548 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c04f07-2576-478b-a56a-c12c3e322119" containerName="maas-api" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606556 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606562 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fd6e4e-8a01-4955-b8a8-a0c907791f05" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606567 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4e3da3-f75a-448b-87eb-281f0d724806" containerName="istio-proxy" Nov 03 09:54:14.606574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.606574 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="185a3ba7-a97f-418a-b243-9647ffe018ed" containerName="maas-api" Nov 03 09:54:14.611286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.611268 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:14.666589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.666551 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-41494e"] Nov 03 09:54:14.713892 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.713861 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-77bkh\"" Nov 03 09:54:14.775590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.775542 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54hl\" (UniqueName: \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\") pod \"test-curl-7384-41494e\" (UID: \"107155c2-c2d0-4708-a1f2-3e2d7f45b24e\") " pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:14.876679 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.876633 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54hl\" (UniqueName: \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\") pod \"test-curl-7384-41494e\" (UID: \"107155c2-c2d0-4708-a1f2-3e2d7f45b24e\") " pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:14.935726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:14.935693 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54hl\" (UniqueName: \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\") pod \"test-curl-7384-41494e\" (UID: \"107155c2-c2d0-4708-a1f2-3e2d7f45b24e\") " pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:15.223438 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:15.223346 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:15.602512 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:15.602483 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-41494e"] Nov 03 09:54:15.624860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:15.624829 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:15.992543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:15.992498 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-41494e" event={"ID":"107155c2-c2d0-4708-a1f2-3e2d7f45b24e","Type":"ContainerStarted","Data":"3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc"} Nov 03 09:54:15.992543 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:15.992546 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-41494e" event={"ID":"107155c2-c2d0-4708-a1f2-3e2d7f45b24e","Type":"ContainerStarted","Data":"f45dcfb5320b5e4ced3a0169bd51a563da3510a17cfb16b66b5ad6853050d534"} Nov 03 09:54:16.110597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:16.110540 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-41494e" podStartSLOduration=2.11052746 podStartE2EDuration="2.11052746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:16.109818251 +0000 UTC m=+4479.777858992" watchObservedRunningTime="2026-11-03 09:54:16.11052746 +0000 UTC m=+4479.778568198" Nov 03 09:54:16.844736 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:16.844701 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:17.569797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.569753 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-41494e"] Nov 03 09:54:17.575783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.575752 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-41494e"] Nov 03 09:54:17.622119 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.622086 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:17.712928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.712893 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-0c243b"] Nov 03 09:54:17.713436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.713422 4543 cpu_manager.go:414] "containerMap: removing container" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" containerName="test-curl-7384-41494e" Nov 03 09:54:17.713486 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.713438 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" containerName="test-curl-7384-41494e" Nov 03 09:54:17.713533 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.713523 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" containerName="test-curl-7384-41494e" Nov 03 09:54:17.716634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.716620 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:17.770895 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.770866 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-0c243b"] Nov 03 09:54:17.808901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.808871 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkhh\" (UniqueName: \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\") pod \"test-curl-7384-0c243b\" (UID: \"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b\") " pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:17.910505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.910473 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkhh\" (UniqueName: \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\") pod \"test-curl-7384-0c243b\" (UID: \"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b\") " pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:17.969946 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:17.969916 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkhh\" (UniqueName: \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\") pod \"test-curl-7384-0c243b\" (UID: \"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b\") " pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:18.004505 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:18.004466 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-41494e" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" containerName="test-curl-7384-41494e" containerID="cri-o://3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc" gracePeriod=2 Nov 03 09:54:18.028892 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:18.028864 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:18.056271 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:18.056236 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-41494e\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:18.402046 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:18.402023 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-0c243b"] Nov 03 09:54:18.403580 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:54:18.403551 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf4f36f5_8f8d_4511_898d_ba0cbcc89e0b.slice/crio-80e46a8f11a628a00e986ec5dfd2bfbd5b01a869a7b7598ace219d23b4b746f0 WatchSource:0}: Error finding container 80e46a8f11a628a00e986ec5dfd2bfbd5b01a869a7b7598ace219d23b4b746f0: Status 404 returned error can't find the container with id 80e46a8f11a628a00e986ec5dfd2bfbd5b01a869a7b7598ace219d23b4b746f0 Nov 03 09:54:19.011950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:19.011914 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-0c243b" event={"ID":"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b","Type":"ContainerStarted","Data":"8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0"} Nov 03 09:54:19.011950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:19.011956 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-0c243b" event={"ID":"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b","Type":"ContainerStarted","Data":"80e46a8f11a628a00e986ec5dfd2bfbd5b01a869a7b7598ace219d23b4b746f0"} Nov 03 09:54:19.127359 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:19.127296 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-0c243b" podStartSLOduration=2.127281187 podStartE2EDuration="2.127281187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:19.126180074 +0000 UTC m=+4482.794220813" watchObservedRunningTime="2026-11-03 09:54:19.127281187 +0000 UTC m=+4482.795321926" Nov 03 09:54:20.222664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.222637 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:20.338233 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.338125 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\" (UniqueName: \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\") pod \"107155c2-c2d0-4708-a1f2-3e2d7f45b24e\" (UID: \"107155c2-c2d0-4708-a1f2-3e2d7f45b24e\") " Nov 03 09:54:20.340399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.340367 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl" pod "107155c2-c2d0-4708-a1f2-3e2d7f45b24e" (UID: "107155c2-c2d0-4708-a1f2-3e2d7f45b24e"). InnerVolumeSpecName "kube-api-access-t54hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:20.439529 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.439493 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t54hl\" (UniqueName: \"kubernetes.io/projected/107155c2-c2d0-4708-a1f2-3e2d7f45b24e-kube-api-access-t54hl\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:20.458361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.458326 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" path="/var/lib/kubelet/pods/107155c2-c2d0-4708-a1f2-3e2d7f45b24e/volumes" Nov 03 09:54:20.492761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.492734 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-0c243b"] Nov 03 09:54:20.496443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:20.496418 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-0c243b"] Nov 03 09:54:21.026312 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.026274 4543 generic.go:356] "Generic (PLEG): container finished" podID="107155c2-c2d0-4708-a1f2-3e2d7f45b24e" containerID="3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc" exitCode=137 Nov 03 09:54:21.026513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.026341 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-41494e" Nov 03 09:54:21.026513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.026377 4543 scope.go:122] "RemoveContainer" containerID="3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc" Nov 03 09:54:21.026630 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.026574 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-0c243b" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" containerName="test-curl-7384-0c243b" containerID="cri-o://8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0" gracePeriod=2 Nov 03 09:54:21.038115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.038091 4543 scope.go:122] "RemoveContainer" containerID="3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc" Nov 03 09:54:21.038654 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:21.038622 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc\": container with ID starting with 3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc not found: ID does not exist" containerID="3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc" Nov 03 09:54:21.038748 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.038667 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc"} err="failed to get container status \"3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc\": rpc error: code = NotFound desc = could not find container \"3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc\": container with ID starting with 3f909810c76a9ce33a2b4325715896421990d704dadea503fcd50be8ff0774cc not found: ID does not exist" Nov 03 09:54:21.078004 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:21.077918 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0c243b\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:21.129332 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:21.129293 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0c243b\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:21.854292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:21.854257 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:23.203179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:23.203154 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:23.267991 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:23.267964 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\" (UniqueName: \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\") pod \"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b\" (UID: \"bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b\") " Nov 03 09:54:23.270210 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:23.270187 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh" pod "bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" (UID: "bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b"). InnerVolumeSpecName "kube-api-access-tdkhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:54:23.368935 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:23.368897 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdkhh\" (UniqueName: \"kubernetes.io/projected/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b-kube-api-access-tdkhh\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:54:24.051103 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.051067 4543 generic.go:356] "Generic (PLEG): container finished" podID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" containerID="8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0" exitCode=137 Nov 03 09:54:24.051302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.051162 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:24.051302 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.051194 4543 scope.go:122] "RemoveContainer" containerID="8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0" Nov 03 09:54:24.061579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.061553 4543 scope.go:122] "RemoveContainer" containerID="8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0" Nov 03 09:54:24.061923 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:24.061897 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0\": container with ID starting with 8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0 not found: ID does not exist" containerID="8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0" Nov 03 09:54:24.062039 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.061933 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0"} err="failed to get container status \"8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0\": rpc error: code = NotFound desc = could not find container \"8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0\": container with ID starting with 8697cc1b16e303aa35087bc06805d2bb7bb191860323dbd76a4618a5bb27b5f0 not found: ID does not exist" Nov 03 09:54:24.112423 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:24.112389 4543 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-0c243b\" is forbidden: User \"system:node:ip-10-0-131-117.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-117.us-west-1.compute.internal' and this object" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" pod="opendatahub/test-curl-7384-0c243b" Nov 03 09:54:24.451922 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.451885 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" path="/var/lib/kubelet/pods/bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b/volumes" Nov 03 09:54:24.671404 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:24.671371 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:25.774096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:25.774060 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:25.995321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:25.995281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:26.142067 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:26.142029 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:27.126374 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:27.126335 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:29.914032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:29.914000 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:30.049044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:30.049013 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:31.212254 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:31.212210 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:32.215311 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:32.215276 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:35.560002 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:54:35.559967 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:54:37.111565 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:37.111534 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:54:37.162977 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:37.162949 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:54:37.282820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:37.282782 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:39.052928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:39.052894 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:39.193901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:39.193870 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:42.379086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:42.379048 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:45.986392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:45.986361 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:46.126041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:46.126001 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:46.976027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:46.975970 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:47.874437 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:47.874404 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:50.822318 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:50.822275 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:50.952382 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:50.952346 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:52.561379 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:52.561350 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:53.637452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:53.637415 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:53.875920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:53.875883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:54.009535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:54.009422 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:54.343877 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:54.343773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:54.680093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:54.680019 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:54.686065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:54.686039 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:54:54.966910 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:54.966798 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:55.105009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:55.104977 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:54:59.438863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:54:59.438825 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:00.168979 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:00.168911 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:01.715361 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:01.715321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:04.435600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:04.435564 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:05.561321 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:55:05.561276 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:55:07.346549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:07.346509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:07.481799 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:07.481767 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:09.570263 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:09.570229 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:09.706504 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:09.706438 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" containerName="main" containerID="cri-o://add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466" gracePeriod=45 Nov 03 09:55:09.793729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:09.793685 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" containerName="main" probeResult="failure" output="Get \"https://10.134.0.140:8000/ready\": dial tcp 10.134.0.140:8000: connect: connection refused" Nov 03 09:55:09.973993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:09.973969 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:55:10.063761 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.063730 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.063961 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.063789 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.063961 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.063819 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.063961 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.063910 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.064188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.063988 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\" (UniqueName: \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.064188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.064038 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\") pod \"3bc4de3c-eae9-4d42-8252-77295e318769\" (UID: \"3bc4de3c-eae9-4d42-8252-77295e318769\") " Nov 03 09:55:10.064188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.064090 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:10.064997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.064475 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.064997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.064960 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:10.065151 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.065074 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:10.066680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.066652 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:55:10.066965 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.066930 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "kube-api-access-xcs8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:10.067297 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.067274 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm" pod "3bc4de3c-eae9-4d42-8252-77295e318769" (UID: "3bc4de3c-eae9-4d42-8252-77295e318769"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:10.097659 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.097630 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:10.166417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.166386 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.166417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.166408 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bc4de3c-eae9-4d42-8252-77295e318769-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.166417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.166417 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.166641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.166427 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bc4de3c-eae9-4d42-8252-77295e318769-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.166641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.166436 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcs8j\" (UniqueName: \"kubernetes.io/projected/3bc4de3c-eae9-4d42-8252-77295e318769-kube-api-access-xcs8j\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:10.318436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.318402 4543 generic.go:356] "Generic (PLEG): container finished" podID="3bc4de3c-eae9-4d42-8252-77295e318769" containerID="add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466" exitCode=0 Nov 03 09:55:10.318626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.318453 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" Nov 03 09:55:10.318626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.318488 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" event={"ID":"3bc4de3c-eae9-4d42-8252-77295e318769","Type":"ContainerDied","Data":"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466"} Nov 03 09:55:10.318626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.318532 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx" event={"ID":"3bc4de3c-eae9-4d42-8252-77295e318769","Type":"ContainerDied","Data":"ec456131cb82a6cdad3373b6b15be2b99fae9cd298a4b0203a38940ab18153a6"} Nov 03 09:55:10.318626 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.318556 4543 scope.go:122] "RemoveContainer" containerID="add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466" Nov 03 09:55:10.327976 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.327958 4543 scope.go:122] "RemoveContainer" containerID="add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466" Nov 03 09:55:10.328241 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:55:10.328221 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466\": container with ID starting with add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466 not found: ID does not exist" containerID="add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466" Nov 03 09:55:10.328306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.328250 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466"} err="failed to get container status \"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466\": rpc error: code = NotFound desc = could not find container \"add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466\": container with ID starting with add4373c7e67f09e5cb04248f4ebfa7a184d6a5c26bd598a0040d3bbcb2a5466 not found: ID does not exist" Nov 03 09:55:10.486936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.486900 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:55:10.537170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:10.537128 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/rate-test-model-9ed110-kserve-7dc9b6674f-9bpqx"] Nov 03 09:55:12.451280 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:12.451250 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" path="/var/lib/kubelet/pods/3bc4de3c-eae9-4d42-8252-77295e318769/volumes" Nov 03 09:55:13.847274 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:13.847230 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:13.983952 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:13.983922 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:14.113355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:14.113321 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:15.898111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:15.898077 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:16.036631 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:16.036594 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:17.412203 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:17.412111 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:17.925306 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:17.925277 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:17.932436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:17.932415 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 09:55:17.932616 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:17.932596 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" containerID="cri-o://568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64" gracePeriod=30 Nov 03 09:55:21.289992 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:21.289953 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:21.518563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:21.518529 4543 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:55:21.518735 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:21.518583 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:55:23.193590 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.193547 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 09:55:23.311082 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.310991 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311107 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311162 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311255 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311300 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311361 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311455 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311545 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311550 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:55:23.311683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311572 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\") pod \"8f87c58b-6851-4213-9570-ea352b8853a4\" (UID: \"8f87c58b-6851-4213-9570-ea352b8853a4\") " Nov 03 09:55:23.311826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311799 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.311826 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.311821 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.312077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.312009 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.312077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.312064 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.312255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.312079 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f87c58b-6851-4213-9570-ea352b8853a4-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.312255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.312093 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.312255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.312177 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.313968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.313949 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:55:23.314078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.314060 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:23.314078 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.314066 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "kube-api-access-65h4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:55:23.314172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.314060 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy" pod "8f87c58b-6851-4213-9570-ea352b8853a4" (UID: "8f87c58b-6851-4213-9570-ea352b8853a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:55:23.402554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.402521 4543 generic.go:356] "Generic (PLEG): container finished" podID="8f87c58b-6851-4213-9570-ea352b8853a4" containerID="568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64" exitCode=0 Nov 03 09:55:23.402771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.402573 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" Nov 03 09:55:23.402771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.402617 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" event={"ID":"8f87c58b-6851-4213-9570-ea352b8853a4","Type":"ContainerDied","Data":"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64"} Nov 03 09:55:23.402771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.402691 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27" event={"ID":"8f87c58b-6851-4213-9570-ea352b8853a4","Type":"ContainerDied","Data":"783cc8bdd7d3878747767fc680502b7e4b6e5fa5af5519775c8a3f5de7e4c8d1"} Nov 03 09:55:23.402771 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.402726 4543 scope.go:122] "RemoveContainer" containerID="568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412883 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412908 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412922 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412935 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f87c58b-6851-4213-9570-ea352b8853a4-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412947 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65h4m\" (UniqueName: \"kubernetes.io/projected/8f87c58b-6851-4213-9570-ea352b8853a4-kube-api-access-65h4m\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.414328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.412961 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f87c58b-6851-4213-9570-ea352b8853a4-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:55:23.415128 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.415103 4543 scope.go:122] "RemoveContainer" containerID="568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64" Nov 03 09:55:23.416539 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:55:23.416501 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64\": container with ID starting with 568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64 not found: ID does not exist" containerID="568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64" Nov 03 09:55:23.416718 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.416692 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64"} err="failed to get container status \"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64\": rpc error: code = NotFound desc = could not find container \"568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64\": container with ID starting with 568b0078fb2d64e7dbf9c8f4f5e5e04e9f9d1e87c00d2c6d19e1dcd636828e64 not found: ID does not exist" Nov 03 09:55:23.572453 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.572374 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 09:55:23.577220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:23.577193 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-fml27"] Nov 03 09:55:24.453072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:24.453033 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" path="/var/lib/kubelet/pods/8f87c58b-6851-4213-9570-ea352b8853a4/volumes" Nov 03 09:55:26.485633 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:26.485600 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:26.837472 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:26.837389 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:32.039220 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:32.039181 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:35.561798 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:55:35.561755 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:55:37.046664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:37.046624 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:37.707356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:37.707324 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:38.700968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:38.700932 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:40.552564 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:40.552531 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:40.991832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:40.991800 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:43.236111 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:43.236075 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:45.632924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:45.632883 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:45.774969 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:45.774929 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:46.054013 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:46.053929 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:48.337420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:48.337386 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:49.703516 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:49.703483 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:49.905426 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:49.905385 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:51.204328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:51.204297 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:51.725823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:51.725783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:52.492896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:52.492864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:52.833198 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:52.833107 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:54.743254 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:54.743202 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:55.684920 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:55.684885 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:55.837107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:55.837075 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:58.651421 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:58.651387 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:58.906100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:58.906019 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:55:59.982722 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:55:59.982665 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:00.113105 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:00.113073 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:00.250322 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:00.250240 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:01.650592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:01.650556 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:02.150223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:02.150194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:02.968389 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:02.968356 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:05.361750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:05.361711 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:05.561365 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:56:05.561328 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:56:06.574652 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:06.574618 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:09.906897 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:09.906847 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:10.441054 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:10.441019 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:10.570157 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:10.570101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:11.038283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:11.038254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:11.167132 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:11.167101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:14.318809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:14.318780 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:15.975425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:15.975391 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:16.153272 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:16.153241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:21.032396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:21.032359 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:24.614729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:24.614689 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:24.742362 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:24.742331 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:24.870956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:24.870872 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:25.495882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.495841 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:56:25.496086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.496064 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" podUID="415508db-2301-4f55-b34e-43734a64d94d" containerName="maas-api" containerID="cri-o://ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0" gracePeriod=30 Nov 03 09:56:25.688894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.688862 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:25.750928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.750839 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:56:25.757109 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.757018 4543 generic.go:356] "Generic (PLEG): container finished" podID="415508db-2301-4f55-b34e-43734a64d94d" containerID="ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0" exitCode=0 Nov 03 09:56:25.757109 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.757091 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" Nov 03 09:56:25.757549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.757089 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" event={"ID":"415508db-2301-4f55-b34e-43734a64d94d","Type":"ContainerDied","Data":"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0"} Nov 03 09:56:25.757549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.757345 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k" event={"ID":"415508db-2301-4f55-b34e-43734a64d94d","Type":"ContainerDied","Data":"7aaf31326b5f935ab5227cc7e493d086310220e739e2473ec6d041bd5e5cba28"} Nov 03 09:56:25.757549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.757377 4543 scope.go:122] "RemoveContainer" containerID="ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0" Nov 03 09:56:25.769282 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.769258 4543 scope.go:122] "RemoveContainer" containerID="ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0" Nov 03 09:56:25.769649 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:56:25.769618 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0\": container with ID starting with ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0 not found: ID does not exist" containerID="ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0" Nov 03 09:56:25.769713 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.769663 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0"} err="failed to get container status \"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0\": rpc error: code = NotFound desc = could not find container \"ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0\": container with ID starting with ce7533b9b88a18fc670011237be692f4b718560f94b98946ed15fe2bff017bc0 not found: ID does not exist" Nov 03 09:56:25.876627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.875592 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\" (UniqueName: \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\") pod \"415508db-2301-4f55-b34e-43734a64d94d\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " Nov 03 09:56:25.876627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.875733 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\") pod \"415508db-2301-4f55-b34e-43734a64d94d\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " Nov 03 09:56:25.876627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.875817 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\") pod \"415508db-2301-4f55-b34e-43734a64d94d\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " Nov 03 09:56:25.876627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.875880 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\") pod \"415508db-2301-4f55-b34e-43734a64d94d\" (UID: \"415508db-2301-4f55-b34e-43734a64d94d\") " Nov 03 09:56:25.878513 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.877359 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca" pod "415508db-2301-4f55-b34e-43734a64d94d" (UID: "415508db-2301-4f55-b34e-43734a64d94d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:25.887432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.887387 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm" pod "415508db-2301-4f55-b34e-43734a64d94d" (UID: "415508db-2301-4f55-b34e-43734a64d94d"). InnerVolumeSpecName "kube-api-access-cnfxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:25.888085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.888053 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls" pod "415508db-2301-4f55-b34e-43734a64d94d" (UID: "415508db-2301-4f55-b34e-43734a64d94d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:25.889984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.889960 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs" pod "415508db-2301-4f55-b34e-43734a64d94d" (UID: "415508db-2301-4f55-b34e-43734a64d94d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:25.977882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.977847 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/415508db-2301-4f55-b34e-43734a64d94d-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:25.977882 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.977881 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:25.978101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.977892 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnfxm\" (UniqueName: \"kubernetes.io/projected/415508db-2301-4f55-b34e-43734a64d94d-kube-api-access-cnfxm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:25.978101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:25.977903 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/415508db-2301-4f55-b34e-43734a64d94d-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:26.188949 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:26.188919 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:56:26.203154 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:26.203112 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-506e6d-7d6979774b-n4k5k"] Nov 03 09:56:26.451429 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:26.451393 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415508db-2301-4f55-b34e-43734a64d94d" path="/var/lib/kubelet/pods/415508db-2301-4f55-b34e-43734a64d94d/volumes" Nov 03 09:56:26.832059 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:26.831972 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:27.678732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:27.678696 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:30.965281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:30.965248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:31.139334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:31.139303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:31.273369 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:31.273284 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:32.224525 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:32.224493 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:33.325561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.325480 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:56:33.325974 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.325697 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" podUID="3534807f-4b41-43ad-b1e8-818679a21968" containerName="maas-api" containerID="cri-o://359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e" gracePeriod=30 Nov 03 09:56:33.363326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.363279 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:33.502853 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.502814 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:33.584166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.584081 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:56:33.759127 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.759099 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\" (UniqueName: \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\") pod \"3534807f-4b41-43ad-b1e8-818679a21968\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " Nov 03 09:56:33.759307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.759205 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\") pod \"3534807f-4b41-43ad-b1e8-818679a21968\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " Nov 03 09:56:33.759307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.759264 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") pod \"3534807f-4b41-43ad-b1e8-818679a21968\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " Nov 03 09:56:33.759307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.759286 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\") pod \"3534807f-4b41-43ad-b1e8-818679a21968\" (UID: \"3534807f-4b41-43ad-b1e8-818679a21968\") " Nov 03 09:56:33.759755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.759722 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca" pod "3534807f-4b41-43ad-b1e8-818679a21968" (UID: "3534807f-4b41-43ad-b1e8-818679a21968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:56:33.761425 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.761395 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt" pod "3534807f-4b41-43ad-b1e8-818679a21968" (UID: "3534807f-4b41-43ad-b1e8-818679a21968"). InnerVolumeSpecName "kube-api-access-mppzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:33.761559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.761474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls" pod "3534807f-4b41-43ad-b1e8-818679a21968" (UID: "3534807f-4b41-43ad-b1e8-818679a21968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:33.761559 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.761498 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs" pod "3534807f-4b41-43ad-b1e8-818679a21968" (UID: "3534807f-4b41-43ad-b1e8-818679a21968"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:33.803447 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.803413 4543 generic.go:356] "Generic (PLEG): container finished" podID="3534807f-4b41-43ad-b1e8-818679a21968" containerID="359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e" exitCode=0 Nov 03 09:56:33.803572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.803471 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" Nov 03 09:56:33.803572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.803503 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" event={"ID":"3534807f-4b41-43ad-b1e8-818679a21968","Type":"ContainerDied","Data":"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e"} Nov 03 09:56:33.803572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.803564 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm" event={"ID":"3534807f-4b41-43ad-b1e8-818679a21968","Type":"ContainerDied","Data":"a4b4f1a897a1e0e3baae2adc3335219af8523e1605fde36e3824925aeef9f82e"} Nov 03 09:56:33.803701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.803586 4543 scope.go:122] "RemoveContainer" containerID="359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e" Nov 03 09:56:33.813340 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.813316 4543 scope.go:122] "RemoveContainer" containerID="359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e" Nov 03 09:56:33.813619 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:56:33.813600 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e\": container with ID starting with 359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e not found: ID does not exist" containerID="359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e" Nov 03 09:56:33.813668 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.813628 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e"} err="failed to get container status \"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e\": rpc error: code = NotFound desc = could not find container \"359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e\": container with ID starting with 359ac982d232cfbb3c55fc630a7a7065cb6ae3937fa35295be453f0c7271221e not found: ID does not exist" Nov 03 09:56:33.860266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.860189 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:33.860266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.860214 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3534807f-4b41-43ad-b1e8-818679a21968-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:33.860266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.860227 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppzt\" (UniqueName: \"kubernetes.io/projected/3534807f-4b41-43ad-b1e8-818679a21968-kube-api-access-mppzt\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:33.860266 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.860237 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3534807f-4b41-43ad-b1e8-818679a21968-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:33.973278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.973246 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:56:33.980715 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:33.980688 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9ed110-7f79b4698b-lz9dm"] Nov 03 09:56:34.451182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:34.451116 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3534807f-4b41-43ad-b1e8-818679a21968" path="/var/lib/kubelet/pods/3534807f-4b41-43ad-b1e8-818679a21968/volumes" Nov 03 09:56:35.005100 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:35.005060 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:35.436005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:35.435905 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:35.567384 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:35.567348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:35.571972 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:56:35.571940 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:56:35.912337 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:35.912296 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:37.980988 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:37.980903 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:38.123808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:38.123775 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:39.533317 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:39.533286 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:40.006832 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:40.006797 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:40.888308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:40.888277 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 09:56:40.893225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:40.893185 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:41.029563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:41.029533 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:41.784436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:41.784402 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:43.096983 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:43.096951 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:43.243914 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:43.243880 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:43.386459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:43.386428 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:44.798048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:44.798018 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:45.305696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:45.305658 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:45.749381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:45.749339 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:46.343319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:46.343276 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:48.725538 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:48.725431 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:48.877257 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:48.877224 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:50.263536 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:50.263493 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:50.740377 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:50.740342 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 09:56:50.789182 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:50.789131 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 09:56:50.814072 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:50.814033 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:50.945562 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:50.945526 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:51.934809 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:51.934773 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:52.723786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:52.723752 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:55.910428 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:55.910384 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" containerID="cri-o://b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9" gracePeriod=45 Nov 03 09:56:56.172730 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.172669 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:56:56.277464 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277434 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.277644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277540 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\" (UniqueName: \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.277644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277572 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.277644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277633 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.277783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277677 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.277783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277703 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\") pod \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\" (UID: \"4681caf5-0ecd-4348-b7a9-eb70ac0e793c\") " Nov 03 09:56:56.278011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277950 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.278011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.277974 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.278011 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.278004 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.278485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.278175 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.278485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.278191 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.278485 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.278205 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.279918 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.279898 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:56:56.280016 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.279995 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:56:56.280116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.280100 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j" pod "4681caf5-0ecd-4348-b7a9-eb70ac0e793c" (UID: "4681caf5-0ecd-4348-b7a9-eb70ac0e793c"). InnerVolumeSpecName "kube-api-access-zq44j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:56:56.379508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.379475 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.379508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.379506 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq44j\" (UniqueName: \"kubernetes.io/projected/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-kube-api-access-zq44j\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.379508 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.379517 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4681caf5-0ecd-4348-b7a9-eb70ac0e793c-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:56:56.932068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.932033 4543 generic.go:356] "Generic (PLEG): container finished" podID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerID="b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9" exitCode=0 Nov 03 09:56:56.932554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.932076 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" Nov 03 09:56:56.932554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.932117 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" event={"ID":"4681caf5-0ecd-4348-b7a9-eb70ac0e793c","Type":"ContainerDied","Data":"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9"} Nov 03 09:56:56.932554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.932180 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp" event={"ID":"4681caf5-0ecd-4348-b7a9-eb70ac0e793c","Type":"ContainerDied","Data":"87f87487aaea0c806ed6276e80d9cf3c93e2a325fddac4279180a2eb259d06ac"} Nov 03 09:56:56.932554 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.932202 4543 scope.go:122] "RemoveContainer" containerID="b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9" Nov 03 09:56:56.941469 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.941451 4543 scope.go:122] "RemoveContainer" containerID="b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9" Nov 03 09:56:56.941738 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:56:56.941719 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9\": container with ID starting with b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9 not found: ID does not exist" containerID="b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9" Nov 03 09:56:56.941814 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:56.941747 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9"} err="failed to get container status \"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9\": rpc error: code = NotFound desc = could not find container \"b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9\": container with ID starting with b071c88a7cbccf12bf0b71f3c8fef7919a5d6ba30f885f299d28b26feac482f9 not found: ID does not exist" Nov 03 09:56:57.045692 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:57.045655 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:57.103107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:57.103077 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 09:56:57.108243 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:57.108218 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/distinct-w2-77207e-kserve-587c5db559-wmffp"] Nov 03 09:56:57.293179 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:57.293079 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:58.381462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:58.381433 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:58.452238 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:58.452176 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" path="/var/lib/kubelet/pods/4681caf5-0ecd-4348-b7a9-eb70ac0e793c/volumes" Nov 03 09:56:58.529165 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:58.529121 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:56:58.731808 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:56:58.731723 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:00.178005 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:00.177966 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:00.985888 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:00.985855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:01.115943 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:01.115913 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:01.847451 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:01.847411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:03.026304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.026260 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 09:57:03.026842 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.026517 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" podUID="af4de6fd-068d-42dc-b681-9ab65b0d5449" containerName="maas-api" containerID="cri-o://9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2" gracePeriod=30 Nov 03 09:57:03.214288 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.214256 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:03.295784 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.295756 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 09:57:03.455048 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455017 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\" (UniqueName: \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\") pod \"af4de6fd-068d-42dc-b681-9ab65b0d5449\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " Nov 03 09:57:03.455275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455064 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\") pod \"af4de6fd-068d-42dc-b681-9ab65b0d5449\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " Nov 03 09:57:03.455275 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455262 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\") pod \"af4de6fd-068d-42dc-b681-9ab65b0d5449\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " Nov 03 09:57:03.455407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455317 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") pod \"af4de6fd-068d-42dc-b681-9ab65b0d5449\" (UID: \"af4de6fd-068d-42dc-b681-9ab65b0d5449\") " Nov 03 09:57:03.455549 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455521 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca" pod "af4de6fd-068d-42dc-b681-9ab65b0d5449" (UID: "af4de6fd-068d-42dc-b681-9ab65b0d5449"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:03.455745 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.455705 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af4de6fd-068d-42dc-b681-9ab65b0d5449-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:03.457546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.457522 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls" pod "af4de6fd-068d-42dc-b681-9ab65b0d5449" (UID: "af4de6fd-068d-42dc-b681-9ab65b0d5449"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:03.457838 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.457816 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2" pod "af4de6fd-068d-42dc-b681-9ab65b0d5449" (UID: "af4de6fd-068d-42dc-b681-9ab65b0d5449"). InnerVolumeSpecName "kube-api-access-mj6l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:03.457899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.457840 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs" pod "af4de6fd-068d-42dc-b681-9ab65b0d5449" (UID: "af4de6fd-068d-42dc-b681-9ab65b0d5449"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:03.556371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.556295 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:03.556371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.556326 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af4de6fd-068d-42dc-b681-9ab65b0d5449-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:03.556371 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.556338 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj6l2\" (UniqueName: \"kubernetes.io/projected/af4de6fd-068d-42dc-b681-9ab65b0d5449-kube-api-access-mj6l2\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:03.970064 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.970030 4543 generic.go:356] "Generic (PLEG): container finished" podID="af4de6fd-068d-42dc-b681-9ab65b0d5449" containerID="9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2" exitCode=0 Nov 03 09:57:03.970268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.970077 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" Nov 03 09:57:03.970268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.970095 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" event={"ID":"af4de6fd-068d-42dc-b681-9ab65b0d5449","Type":"ContainerDied","Data":"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2"} Nov 03 09:57:03.970268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.970154 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl" event={"ID":"af4de6fd-068d-42dc-b681-9ab65b0d5449","Type":"ContainerDied","Data":"f96f8dff1ea861ad13c8d8c98662b7a85fd3f158d6c908e4bee90ab8d0e35462"} Nov 03 09:57:03.970268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.970176 4543 scope.go:122] "RemoveContainer" containerID="9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2" Nov 03 09:57:03.986181 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.986120 4543 scope.go:122] "RemoveContainer" containerID="9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2" Nov 03 09:57:03.986656 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:03.986631 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2\": container with ID starting with 9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2 not found: ID does not exist" containerID="9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2" Nov 03 09:57:03.986750 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:03.986670 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2"} err="failed to get container status \"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2\": rpc error: code = NotFound desc = could not find container \"9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2\": container with ID starting with 9246881f8fefc65a574e536f1d1d24bc50c243310e9554fd71a92320e7c211b2 not found: ID does not exist" Nov 03 09:57:04.138840 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.138805 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 09:57:04.145285 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.145262 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-77207e-dbb497ddd-kf4hl"] Nov 03 09:57:04.180665 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.180625 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:04.450687 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.450658 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4de6fd-068d-42dc-b681-9ab65b0d5449" path="/var/lib/kubelet/pods/af4de6fd-068d-42dc-b681-9ab65b0d5449/volumes" Nov 03 09:57:04.669644 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.669613 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:04.815009 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.814929 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:04.952561 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:04.952531 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:05.561262 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:05.561230 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:57:05.761208 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.761168 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" containerID="cri-o://799ba95f65474fc1fcd9ef3e7e0b47bfdc45f22e0afaa17545d1d392cc231dfe" gracePeriod=45 Nov 03 09:57:05.809732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.809670 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" containerID="cri-o://ef338a02a5b2e24d301243d0abdd0c620b8707c723128bd54eee00e0f867f8e5" gracePeriod=45 Nov 03 09:57:05.985546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.985043 4543 generic.go:356] "Generic (PLEG): container finished" podID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerID="ef338a02a5b2e24d301243d0abdd0c620b8707c723128bd54eee00e0f867f8e5" exitCode=0 Nov 03 09:57:05.985546 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.985233 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" event={"ID":"51cc7f98-70b1-4077-bcb5-3e19ab21ba78","Type":"ContainerDied","Data":"ef338a02a5b2e24d301243d0abdd0c620b8707c723128bd54eee00e0f867f8e5"} Nov 03 09:57:05.987170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.987068 4543 generic.go:356] "Generic (PLEG): container finished" podID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerID="799ba95f65474fc1fcd9ef3e7e0b47bfdc45f22e0afaa17545d1d392cc231dfe" exitCode=0 Nov 03 09:57:05.987170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:05.987167 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" event={"ID":"de7e967f-e4a7-4a65-bf67-53245d5cce74","Type":"ContainerDied","Data":"799ba95f65474fc1fcd9ef3e7e0b47bfdc45f22e0afaa17545d1d392cc231dfe"} Nov 03 09:57:06.069573 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.069551 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:57:06.097204 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.097129 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:57:06.183852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.183809 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.183866 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.183901 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\" (UniqueName: \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.183929 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.184049 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.183965 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\" (UniqueName: \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184101 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.184341 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184210 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.184517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184470 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.184579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184513 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184574 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\") pod \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\" (UID: \"51cc7f98-70b1-4077-bcb5-3e19ab21ba78\") " Nov 03 09:57:06.184690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184658 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\" (UniqueName: \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.184754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184672 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.184754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184730 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.184754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184740 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.184962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\" (UniqueName: \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.184962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184788 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.184962 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.184808 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\") pod \"de7e967f-e4a7-4a65-bf67-53245d5cce74\" (UID: \"de7e967f-e4a7-4a65-bf67-53245d5cce74\") " Nov 03 09:57:06.185235 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185209 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.185334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185229 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.185334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185257 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.185334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185270 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.185334 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185284 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.185588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.185493 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.188681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.188593 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:06.188681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.188637 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.188924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.188894 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:06.189316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.189289 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "kube-api-access-fwpgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:06.189721 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.189698 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs" pod "51cc7f98-70b1-4077-bcb5-3e19ab21ba78" (UID: "51cc7f98-70b1-4077-bcb5-3e19ab21ba78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:06.189916 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.189896 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww" pod "de7e967f-e4a7-4a65-bf67-53245d5cce74" (UID: "de7e967f-e4a7-4a65-bf67-53245d5cce74"). InnerVolumeSpecName "kube-api-access-8s8ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:06.286378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286348 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286378 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286378 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwpgr\" (UniqueName: \"kubernetes.io/projected/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-kube-api-access-fwpgr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286391 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51cc7f98-70b1-4077-bcb5-3e19ab21ba78-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286403 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de7e967f-e4a7-4a65-bf67-53245d5cce74-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286414 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286426 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8ww\" (UniqueName: \"kubernetes.io/projected/de7e967f-e4a7-4a65-bf67-53245d5cce74-kube-api-access-8s8ww\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.286589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.286437 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de7e967f-e4a7-4a65-bf67-53245d5cce74-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:06.322258 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.322225 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:06.790860 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.790826 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:57:06.791367 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.791087 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" containerID="cri-o://0694b2ae756be08d04f4cc52ad890f6db10f3a7640cf3fa4fb26320ba865d2b3" gracePeriod=30 Nov 03 09:57:06.797524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.797499 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:06.981101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.981069 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:06.993747 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.993706 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" event={"ID":"51cc7f98-70b1-4077-bcb5-3e19ab21ba78","Type":"ContainerDied","Data":"9106cdffcc8560cc7268bc56e485fd8ebddb783ffd968daaa856b50652dde848"} Nov 03 09:57:06.993903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.993758 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q" Nov 03 09:57:06.993903 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.993775 4543 scope.go:122] "RemoveContainer" containerID="ef338a02a5b2e24d301243d0abdd0c620b8707c723128bd54eee00e0f867f8e5" Nov 03 09:57:06.995150 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.995118 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" Nov 03 09:57:06.995239 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:06.995112 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b" event={"ID":"de7e967f-e4a7-4a65-bf67-53245d5cce74","Type":"ContainerDied","Data":"950fb9e0999f92defd2d1596a6f2a0f1ba306093a0502caaa88bf8be6b8b3ecb"} Nov 03 09:57:07.003123 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.003106 4543 scope.go:122] "RemoveContainer" containerID="799ba95f65474fc1fcd9ef3e7e0b47bfdc45f22e0afaa17545d1d392cc231dfe" Nov 03 09:57:07.161435 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.161402 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 09:57:07.165868 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.165841 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/simulator-w0-910d50-kserve-6d7d59b498-vqb9q"] Nov 03 09:57:07.240393 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.240356 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:07.281480 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.281449 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 09:57:07.286392 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:07.286366 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/premium-w0-910d50-kserve-76cdbd64d7-n6v5b"] Nov 03 09:57:08.452387 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:08.452358 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" path="/var/lib/kubelet/pods/51cc7f98-70b1-4077-bcb5-3e19ab21ba78/volumes" Nov 03 09:57:08.452778 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:08.452770 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" path="/var/lib/kubelet/pods/de7e967f-e4a7-4a65-bf67-53245d5cce74/volumes" Nov 03 09:57:10.530852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:10.530812 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:10.781805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:10.781722 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:11.123997 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:11.123966 4543 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:57:11.124200 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:11.124026 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:57:12.024680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.024648 4543 generic.go:356] "Generic (PLEG): container finished" podID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerID="0694b2ae756be08d04f4cc52ad890f6db10f3a7640cf3fa4fb26320ba865d2b3" exitCode=0 Nov 03 09:57:12.025215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.024702 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerDied","Data":"0694b2ae756be08d04f4cc52ad890f6db10f3a7640cf3fa4fb26320ba865d2b3"} Nov 03 09:57:12.025215 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.024793 4543 scope.go:122] "RemoveContainer" containerID="d68a4ff8f719418e8f2ac5f84538480e6889a7578089c10dec2d99e31f77ad70" Nov 03 09:57:12.061498 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.061448 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:57:12.214335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.214296 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 09:57:12.247733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247703 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.247907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247775 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.247907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.247907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247893 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247920 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.247979 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248068 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248007 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248058 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248265 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:12.248333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248283 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:12.248333 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248300 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:12.248499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248394 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\") pod \"1fe29adb-8c7b-4eee-8a75-5502439d5978\" (UID: \"1fe29adb-8c7b-4eee-8a75-5502439d5978\") " Nov 03 09:57:12.248499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248405 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:12.248647 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248619 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:12.248844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248827 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe29adb-8c7b-4eee-8a75-5502439d5978-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.248912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248852 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.248912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248868 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.248912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248882 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.248912 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.248896 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.250890 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.250852 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:12.251255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.251226 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:57:12.251664 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.251642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "kube-api-access-dp4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:12.251770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.251749 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token" pod "1fe29adb-8c7b-4eee-8a75-5502439d5978" (UID: "1fe29adb-8c7b-4eee-8a75-5502439d5978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:12.315115 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.315025 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:12.350432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.350402 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.350432 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.350431 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.350641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.350444 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4bh\" (UniqueName: \"kubernetes.io/projected/1fe29adb-8c7b-4eee-8a75-5502439d5978-kube-api-access-dp4bh\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.350641 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.350460 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe29adb-8c7b-4eee-8a75-5502439d5978-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:12.449984 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:12.449955 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:13.031253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:13.031221 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" Nov 03 09:57:13.031253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:13.031230 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95" event={"ID":"1fe29adb-8c7b-4eee-8a75-5502439d5978","Type":"ContainerDied","Data":"391a5421b794cf239070b063202b10447dad6e5017528d4fdbf59ec9ad872fcc"} Nov 03 09:57:13.031769 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:13.031291 4543 scope.go:122] "RemoveContainer" containerID="0694b2ae756be08d04f4cc52ad890f6db10f3a7640cf3fa4fb26320ba865d2b3" Nov 03 09:57:13.229827 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:13.229788 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:57:13.280844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:13.280811 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-bnk95"] Nov 03 09:57:14.323995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.323958 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:14.450107 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.450073 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" path="/var/lib/kubelet/pods/1fe29adb-8c7b-4eee-8a75-5502439d5978/volumes" Nov 03 09:57:14.670779 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.670750 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:14.898998 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.898967 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 09:57:14.899205 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.899183 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" podUID="f77e93d9-113b-41ce-b575-796a92e08e82" containerName="maas-api" containerID="cri-o://105b25208a3301899740416451ac3e28ab0bf919e24e6c6df7495526b3c25f90" gracePeriod=30 Nov 03 09:57:14.990534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:14.990463 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:15.048113 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.048081 4543 generic.go:356] "Generic (PLEG): container finished" podID="f77e93d9-113b-41ce-b575-796a92e08e82" containerID="105b25208a3301899740416451ac3e28ab0bf919e24e6c6df7495526b3c25f90" exitCode=0 Nov 03 09:57:15.048299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.048163 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" event={"ID":"f77e93d9-113b-41ce-b575-796a92e08e82","Type":"ContainerDied","Data":"105b25208a3301899740416451ac3e28ab0bf919e24e6c6df7495526b3c25f90"} Nov 03 09:57:15.153223 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.153200 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 09:57:15.277774 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.277661 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\" (UniqueName: \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\") pod \"f77e93d9-113b-41ce-b575-796a92e08e82\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " Nov 03 09:57:15.277966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.277773 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\") pod \"f77e93d9-113b-41ce-b575-796a92e08e82\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " Nov 03 09:57:15.277966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.277819 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\") pod \"f77e93d9-113b-41ce-b575-796a92e08e82\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " Nov 03 09:57:15.277966 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.277894 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") pod \"f77e93d9-113b-41ce-b575-796a92e08e82\" (UID: \"f77e93d9-113b-41ce-b575-796a92e08e82\") " Nov 03 09:57:15.278355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.278330 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca" pod "f77e93d9-113b-41ce-b575-796a92e08e82" (UID: "f77e93d9-113b-41ce-b575-796a92e08e82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:15.280654 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.280579 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs" pod "f77e93d9-113b-41ce-b575-796a92e08e82" (UID: "f77e93d9-113b-41ce-b575-796a92e08e82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:15.282864 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.282833 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h" pod "f77e93d9-113b-41ce-b575-796a92e08e82" (UID: "f77e93d9-113b-41ce-b575-796a92e08e82"). InnerVolumeSpecName "kube-api-access-jdg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:15.297396 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.295293 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls" pod "f77e93d9-113b-41ce-b575-796a92e08e82" (UID: "f77e93d9-113b-41ce-b575-796a92e08e82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:15.378852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.378818 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:15.378852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.378846 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdg7h\" (UniqueName: \"kubernetes.io/projected/f77e93d9-113b-41ce-b575-796a92e08e82-kube-api-access-jdg7h\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:15.378852 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.378856 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77e93d9-113b-41ce-b575-796a92e08e82-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:15.379352 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.378865 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f77e93d9-113b-41ce-b575-796a92e08e82-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:15.733216 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:15.733158 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:16.054537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.054447 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" event={"ID":"f77e93d9-113b-41ce-b575-796a92e08e82","Type":"ContainerDied","Data":"efe9a9e091566b549d7c5e2c3fdf157d791e5faac16f01e7388ff67c69116675"} Nov 03 09:57:16.054537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.054469 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl" Nov 03 09:57:16.054537 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.054511 4543 scope.go:122] "RemoveContainer" containerID="105b25208a3301899740416451ac3e28ab0bf919e24e6c6df7495526b3c25f90" Nov 03 09:57:16.221968 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.221935 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 09:57:16.272414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.272352 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-910d50-677688d75c-qglcl"] Nov 03 09:57:16.450819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.450787 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77e93d9-113b-41ce-b575-796a92e08e82" path="/var/lib/kubelet/pods/f77e93d9-113b-41ce-b575-796a92e08e82/volumes" Nov 03 09:57:16.817906 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:16.817821 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:17.239925 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:17.239862 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:17.372753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:17.372715 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:20.004094 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:20.004051 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:20.503114 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:20.503084 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:20.973364 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:20.973328 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:21.735700 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:21.735666 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:22.015436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:22.015350 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:22.419994 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:22.419958 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:22.556848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:22.556813 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:23.306184 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:23.306127 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:23.434035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:23.434001 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:24.769287 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:24.769254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:26.538172 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:26.538117 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:26.665244 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:26.665206 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:27.184955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.184921 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:27.235328 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.235288 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" containerID="cri-o://3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3" gracePeriod=45 Nov 03 09:57:27.482636 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.482613 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.607772 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\" (UniqueName: \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.607849 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.607928 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.607997 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.608069 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.608117 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\") pod \"ca58f7f4-98c5-4061-9ce3-95913f05e795\" (UID: \"ca58f7f4-98c5-4061-9ce3-95913f05e795\") " Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.608455 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.608733 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.608469 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.609701 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.609013 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.611445 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.611399 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:27.611623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.611489 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "kube-api-access-7ptm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:27.611623 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.611526 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm" pod "ca58f7f4-98c5-4061-9ce3-95913f05e795" (UID: "ca58f7f4-98c5-4061-9ce3-95913f05e795"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:27.710000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.709956 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.710000 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.710000 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.710314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.710015 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.710314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.710029 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptm2\" (UniqueName: \"kubernetes.io/projected/ca58f7f4-98c5-4061-9ce3-95913f05e795-kube-api-access-7ptm2\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.710314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.710043 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58f7f4-98c5-4061-9ce3-95913f05e795-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:27.710314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:27.710056 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58f7f4-98c5-4061-9ce3-95913f05e795-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:28.117956 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.117921 4543 generic.go:356] "Generic (PLEG): container finished" podID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerID="3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3" exitCode=0 Nov 03 09:57:28.118170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.117966 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" Nov 03 09:57:28.118170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.118004 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" event={"ID":"ca58f7f4-98c5-4061-9ce3-95913f05e795","Type":"ContainerDied","Data":"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3"} Nov 03 09:57:28.118170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.118045 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr" event={"ID":"ca58f7f4-98c5-4061-9ce3-95913f05e795","Type":"ContainerDied","Data":"0e7900f0dc8ae8bac65ee15cc2eded4edde0f6a962c76e74b56ccd40284533ff"} Nov 03 09:57:28.118170 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.118065 4543 scope.go:122] "RemoveContainer" containerID="3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3" Nov 03 09:57:28.127874 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.127857 4543 scope.go:122] "RemoveContainer" containerID="3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3" Nov 03 09:57:28.128207 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:28.128182 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3\": container with ID starting with 3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3 not found: ID does not exist" containerID="3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3" Nov 03 09:57:28.128269 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.128219 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3"} err="failed to get container status \"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3\": rpc error: code = NotFound desc = could not find container \"3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3\": container with ID starting with 3466c73ef1eab26ddc9bbfd045df34a0be2f449e5eca3a2d588642edc328d6e3 not found: ID does not exist" Nov 03 09:57:28.285496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.285462 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 09:57:28.335990 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.335957 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-w3-98d5dd-kserve-5b8c79d59b-76fqr"] Nov 03 09:57:28.450879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.450802 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" path="/var/lib/kubelet/pods/ca58f7f4-98c5-4061-9ce3-95913f05e795/volumes" Nov 03 09:57:28.851634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:28.851554 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:29.446299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:29.446254 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:29.577542 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:29.577509 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:30.116358 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:30.116283 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:31.633240 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:31.633198 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:31.764155 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:31.764115 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:33.196126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:33.196097 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:33.323156 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:33.323055 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:33.663901 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:33.663865 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:33.796448 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:33.796416 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:34.186314 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:34.186268 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:34.195032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:34.195010 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 09:57:34.195252 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:34.195232 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" containerID="cri-o://b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539" gracePeriod=30 Nov 03 09:57:35.560942 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:35.560910 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:57:36.055162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:36.055047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:36.970986 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:36.970945 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:37.279283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.279199 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 09:57:37.279479 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.279436 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" podUID="40b95d18-8494-4c60-a900-a7eeb091e82d" containerName="maas-api" containerID="cri-o://6153e495b3d7f3697a9fc1b9e08b88d96258bfdccaebbb2bde028c5749685ab4" gracePeriod=30 Nov 03 09:57:37.369511 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.369476 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:37.526634 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.526610 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 09:57:37.604077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604003 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") pod \"40b95d18-8494-4c60-a900-a7eeb091e82d\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " Nov 03 09:57:37.604077 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604047 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\") pod \"40b95d18-8494-4c60-a900-a7eeb091e82d\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " Nov 03 09:57:37.604296 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604249 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\") pod \"40b95d18-8494-4c60-a900-a7eeb091e82d\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " Nov 03 09:57:37.604363 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604313 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\" (UniqueName: \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\") pod \"40b95d18-8494-4c60-a900-a7eeb091e82d\" (UID: \"40b95d18-8494-4c60-a900-a7eeb091e82d\") " Nov 03 09:57:37.604593 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604553 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca" pod "40b95d18-8494-4c60-a900-a7eeb091e82d" (UID: "40b95d18-8494-4c60-a900-a7eeb091e82d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:37.604939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.604646 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b95d18-8494-4c60-a900-a7eeb091e82d-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:37.606351 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.606324 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs" pod "40b95d18-8494-4c60-a900-a7eeb091e82d" (UID: "40b95d18-8494-4c60-a900-a7eeb091e82d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:37.606452 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.606352 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls" pod "40b95d18-8494-4c60-a900-a7eeb091e82d" (UID: "40b95d18-8494-4c60-a900-a7eeb091e82d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:57:37.606507 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.606443 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l" pod "40b95d18-8494-4c60-a900-a7eeb091e82d" (UID: "40b95d18-8494-4c60-a900-a7eeb091e82d"). InnerVolumeSpecName "kube-api-access-85r4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:37.705619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.705580 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:37.705619 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.705617 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b95d18-8494-4c60-a900-a7eeb091e82d-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:37.705831 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.705634 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85r4l\" (UniqueName: \"kubernetes.io/projected/40b95d18-8494-4c60-a900-a7eeb091e82d-kube-api-access-85r4l\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:37.715651 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:37.715622 4543 scope.go:122] "RemoveContainer" containerID="6153e495b3d7f3697a9fc1b9e08b88d96258bfdccaebbb2bde028c5749685ab4" Nov 03 09:57:38.035680 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.035647 4543 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 09:57:38.036085 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.035708 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 09:57:38.174755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.174720 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" event={"ID":"40b95d18-8494-4c60-a900-a7eeb091e82d","Type":"ContainerDied","Data":"6153e495b3d7f3697a9fc1b9e08b88d96258bfdccaebbb2bde028c5749685ab4"} Nov 03 09:57:38.174928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.174764 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" Nov 03 09:57:38.174928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.174779 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm" event={"ID":"40b95d18-8494-4c60-a900-a7eeb091e82d","Type":"ContainerDied","Data":"398487228fdec36b23bea0cdb1d9e4c1c1707584d8d50431af8ce3e61ad4fbc8"} Nov 03 09:57:38.405283 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.405242 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 09:57:38.410413 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.410382 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-98d5dd-6d8b8bbfb7-vtjdm"] Nov 03 09:57:38.450466 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:38.450424 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b95d18-8494-4c60-a900-a7eeb091e82d" path="/var/lib/kubelet/pods/40b95d18-8494-4c60-a900-a7eeb091e82d/volumes" Nov 03 09:57:39.465646 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.465581 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 09:57:39.634783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634742 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.634951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634831 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.634951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634885 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.634951 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634933 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634957 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.634999 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635091 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635068 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635107 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635171 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\") pod \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\" (UID: \"d5717801-8479-4b75-8f4f-4ee3b9c78f94\") " Nov 03 09:57:39.635270 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635215 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:39.635436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635266 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:39.635436 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635300 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:39.635556 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635528 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:39.635556 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635536 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:39.635683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635627 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.635683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635640 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.635683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635649 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.635683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635662 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.635683 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.635676 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.637409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.637388 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:39.637517 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.637494 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "kube-api-access-4qqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:39.637653 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.637639 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:39.637823 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.637809 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo" pod "d5717801-8479-4b75-8f4f-4ee3b9c78f94" (UID: "d5717801-8479-4b75-8f4f-4ee3b9c78f94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:57:39.736690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.736609 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.736690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.736644 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqgq\" (UniqueName: \"kubernetes.io/projected/d5717801-8479-4b75-8f4f-4ee3b9c78f94-kube-api-access-4qqgq\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.736690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.736656 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:39.736690 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:39.736664 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5717801-8479-4b75-8f4f-4ee3b9c78f94-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:40.185381 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.185343 4543 generic.go:356] "Generic (PLEG): container finished" podID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerID="b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539" exitCode=0 Nov 03 09:57:40.185589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.185395 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" Nov 03 09:57:40.185589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.185430 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerDied","Data":"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539"} Nov 03 09:57:40.185589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.185485 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7" event={"ID":"d5717801-8479-4b75-8f4f-4ee3b9c78f94","Type":"ContainerDied","Data":"51102261032a0b352493351bdebc31fe28d1a1b60eb4df0d2caea655ac004475"} Nov 03 09:57:40.185589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.185513 4543 scope.go:122] "RemoveContainer" containerID="b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539" Nov 03 09:57:40.196993 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.196968 4543 scope.go:122] "RemoveContainer" containerID="f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b" Nov 03 09:57:40.205286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.205264 4543 scope.go:122] "RemoveContainer" containerID="b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539" Nov 03 09:57:40.205563 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:40.205544 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539\": container with ID starting with b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539 not found: ID does not exist" containerID="b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539" Nov 03 09:57:40.205625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.205573 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539"} err="failed to get container status \"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539\": rpc error: code = NotFound desc = could not find container \"b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539\": container with ID starting with b6b9c8a26e3b768e6b2c6e5748da13bf979f13127f77e46eed7ef08a23cc3539 not found: ID does not exist" Nov 03 09:57:40.205625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.205592 4543 scope.go:122] "RemoveContainer" containerID="f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b" Nov 03 09:57:40.205842 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:40.205828 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b\": container with ID starting with f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b not found: ID does not exist" containerID="f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b" Nov 03 09:57:40.205899 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.205845 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b"} err="failed to get container status \"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b\": rpc error: code = NotFound desc = could not find container \"f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b\": container with ID starting with f8b266ebf64196cdd62ab0f1bf2d9c03c4e5de334d76ec541dc4fa54d28fc35b not found: ID does not exist" Nov 03 09:57:40.357076 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.357042 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 09:57:40.407697 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.407663 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-q7pb7"] Nov 03 09:57:40.450279 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:40.450203 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" path="/var/lib/kubelet/pods/d5717801-8479-4b75-8f4f-4ee3b9c78f94/volumes" Nov 03 09:57:42.929739 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:42.929707 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:43.124964 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:43.124935 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:43.674268 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:43.674234 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:43.807066 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:43.807039 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:50.243812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:50.243774 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:50.606836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:50.606725 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:50.736927 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:50.736892 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:50.869199 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:50.869168 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:53.157185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:53.157149 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:53.626524 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:53.626484 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:53.640995 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:53.640961 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 09:57:53.641249 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:53.641225 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" containerID="cri-o://4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca" gracePeriod=30 Nov 03 09:57:55.345234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:55.345206 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 09:57:55.610133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:55.610063 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 09:57:57.259491 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:57.259454 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:57:57.591402 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:57.591300 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 09:57:58.724526 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:58.724496 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 09:57:58.901688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:58.901666 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 09:57:59.017110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017018 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017058 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017110 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017098 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017123 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017212 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017268 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017296 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017347 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017385 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\") pod \"f4fc4824-232a-4853-8f9f-9e183c2159c5\" (UID: \"f4fc4824-232a-4853-8f9f-9e183c2159c5\") " Nov 03 09:57:59.017443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017430 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:59.017770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017509 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:59.017770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017635 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:57:59.017770 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017735 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:59.017896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017784 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.017896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017797 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4fc4824-232a-4853-8f9f-9e183c2159c5-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.017896 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.017811 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.018036 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.018011 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:59.019499 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.019474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "kube-api-access-4w6z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:59.019753 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.019731 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:57:59.019967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.019948 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:57:59.019967 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.019951 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy" pod "f4fc4824-232a-4853-8f9f-9e183c2159c5" (UID: "f4fc4824-232a-4853-8f9f-9e183c2159c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:57:59.085167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.085098 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 09:57:59.118732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118693 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.118732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118721 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.118732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118731 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6z7\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-kube-api-access-4w6z7\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.118950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118741 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.118950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118749 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4fc4824-232a-4853-8f9f-9e183c2159c5-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.118950 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.118758 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4fc4824-232a-4853-8f9f-9e183c2159c5-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:57:59.275588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.275493 4543 generic.go:356] "Generic (PLEG): container finished" podID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerID="4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca" exitCode=0 Nov 03 09:57:59.275588 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.275551 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" Nov 03 09:57:59.275829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.275576 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerDied","Data":"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca"} Nov 03 09:57:59.275829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.275646 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq" event={"ID":"f4fc4824-232a-4853-8f9f-9e183c2159c5","Type":"ContainerDied","Data":"5893c9796373b5baa412607bf4013b00c813c626d2dd420bd3594a0e7f55578d"} Nov 03 09:57:59.275829 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.275677 4543 scope.go:122] "RemoveContainer" containerID="4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca" Nov 03 09:57:59.289521 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.289504 4543 scope.go:122] "RemoveContainer" containerID="d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611" Nov 03 09:57:59.298162 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.298121 4543 scope.go:122] "RemoveContainer" containerID="4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca" Nov 03 09:57:59.298438 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:59.298415 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca\": container with ID starting with 4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca not found: ID does not exist" containerID="4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca" Nov 03 09:57:59.298598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.298446 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca"} err="failed to get container status \"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca\": rpc error: code = NotFound desc = could not find container \"4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca\": container with ID starting with 4cf20c003fb36289d52684917cabce7dae392c9c14e89ac3fb0bd8165bba6fca not found: ID does not exist" Nov 03 09:57:59.298598 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.298463 4543 scope.go:122] "RemoveContainer" containerID="d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611" Nov 03 09:57:59.298732 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:57:59.298703 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611\": container with ID starting with d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611 not found: ID does not exist" containerID="d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611" Nov 03 09:57:59.298806 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.298738 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611"} err="failed to get container status \"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611\": rpc error: code = NotFound desc = could not find container \"d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611\": container with ID starting with d9f83d0fb713ceb2025d618c2a2c3850d46f27143350b98271af6db94726c611 not found: ID does not exist" Nov 03 09:57:59.444347 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.444312 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 09:57:59.494754 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:57:59.494722 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-5tqfq"] Nov 03 09:58:00.449936 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:00.449902 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" path="/var/lib/kubelet/pods/f4fc4824-232a-4853-8f9f-9e183c2159c5/volumes" Nov 03 09:58:03.532848 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:03.532810 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:03.673741 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:03.673708 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:05.560651 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:58:05.560609 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:58:09.916907 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:09.916877 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 09:58:10.147420 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:10.147393 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 09:58:11.129299 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:11.129262 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:15.015286 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:15.015248 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:15.644674 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:15.644632 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:16.216824 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.216788 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:58:16.217253 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217239 4543 cpu_manager.go:414] "containerMap: removing container" podUID="af4de6fd-068d-42dc-b681-9ab65b0d5449" containerName="maas-api" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217255 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4de6fd-068d-42dc-b681-9ab65b0d5449" containerName="maas-api" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217263 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" containerName="main" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217268 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" containerName="main" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217276 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217285 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217296 4543 cpu_manager.go:414] "containerMap: removing container" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" Nov 03 09:58:16.217304 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217301 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217313 4543 cpu_manager.go:414] "containerMap: removing container" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" containerName="test-curl-7384-0c243b" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217317 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" containerName="test-curl-7384-0c243b" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217331 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f77e93d9-113b-41ce-b575-796a92e08e82" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217335 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77e93d9-113b-41ce-b575-796a92e08e82" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217341 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217346 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217352 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217358 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217373 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217379 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217385 4543 cpu_manager.go:414] "containerMap: removing container" podUID="415508db-2301-4f55-b34e-43734a64d94d" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217389 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415508db-2301-4f55-b34e-43734a64d94d" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217395 4543 cpu_manager.go:414] "containerMap: removing container" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217399 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217408 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217412 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217417 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217421 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217428 4543 cpu_manager.go:414] "containerMap: removing container" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217432 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217440 4543 cpu_manager.go:414] "containerMap: removing container" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217448 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217459 4543 cpu_manager.go:414] "containerMap: removing container" podUID="40b95d18-8494-4c60-a900-a7eeb091e82d" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217463 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b95d18-8494-4c60-a900-a7eeb091e82d" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217470 4543 cpu_manager.go:414] "containerMap: removing container" podUID="3534807f-4b41-43ad-b1e8-818679a21968" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217475 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3534807f-4b41-43ad-b1e8-818679a21968" containerName="maas-api" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217480 4543 cpu_manager.go:414] "containerMap: removing container" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" Nov 03 09:58:16.217522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217484 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217553 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217563 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f87c58b-6851-4213-9570-ea352b8853a4" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217568 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217573 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5717801-8479-4b75-8f4f-4ee3b9c78f94" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217579 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b95d18-8494-4c60-a900-a7eeb091e82d" containerName="maas-api" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217584 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="4681caf5-0ecd-4348-b7a9-eb70ac0e793c" containerName="main" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217589 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3534807f-4b41-43ad-b1e8-818679a21968" containerName="maas-api" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217596 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4de6fd-068d-42dc-b681-9ab65b0d5449" containerName="maas-api" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217604 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc4de3c-eae9-4d42-8252-77295e318769" containerName="main" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217612 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217618 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7e967f-e4a7-4a65-bf67-53245d5cce74" containerName="main" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217624 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf4f36f5-8f8d-4511-898d-ba0cbcc89e0b" containerName="test-curl-7384-0c243b" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217630 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77e93d9-113b-41ce-b575-796a92e08e82" containerName="maas-api" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217634 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fc4824-232a-4853-8f9f-9e183c2159c5" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217638 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe29adb-8c7b-4eee-8a75-5502439d5978" containerName="istio-proxy" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217644 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca58f7f4-98c5-4061-9ce3-95913f05e795" containerName="main" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217649 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="415508db-2301-4f55-b34e-43734a64d94d" containerName="maas-api" Nov 03 09:58:16.218375 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.217653 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cc7f98-70b1-4077-bcb5-3e19ab21ba78" containerName="main" Nov 03 09:58:16.222070 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.222052 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.276398 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.276368 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 09:58:16.296108 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.296081 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a3965d89-openshift-default-dockercfg-xwvcw\"" Nov 03 09:58:16.307627 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.307599 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:16.328844 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.328800 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:58:16.372924 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.372891 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.372946 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373041 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373130 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373089 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373155 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5xtq\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373203 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373254 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373287 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.373321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.373305 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.473938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.473858 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.473938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.473903 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.473935 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.473969 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.473993 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5xtq\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474020 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474059 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474102 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474124 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474333 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474357 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474499 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474572 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474525 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.474911 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.474888 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.476929 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.476902 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.477028 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.477011 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.533126 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.533092 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.533383 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.533361 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5xtq\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\") pod \"e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.535055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.535034 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:16.704731 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.704697 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:16.965863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.965836 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:58:16.967805 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:58:16.967777 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06416e7c_2d65_4444_bd96_bcfbf54beb4d.slice/crio-827d970415cc6db82f1c939795b2b8d7150b5955163621232dc4e9cadf0b4998 WatchSource:0}: Error finding container 827d970415cc6db82f1c939795b2b8d7150b5955163621232dc4e9cadf0b4998: Status 404 returned error can't find the container with id 827d970415cc6db82f1c939795b2b8d7150b5955163621232dc4e9cadf0b4998 Nov 03 09:58:16.969938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.969905 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:16.970041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.969986 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:16.970041 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:16.970030 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:17.362018 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:17.361993 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerStarted","Data":"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f"} Nov 03 09:58:17.362434 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:17.362042 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerStarted","Data":"827d970415cc6db82f1c939795b2b8d7150b5955163621232dc4e9cadf0b4998"} Nov 03 09:58:17.483934 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:17.483869 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podStartSLOduration=1.483849893 podStartE2EDuration="1.483849893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:58:17.482292682 +0000 UTC m=+4721.150333421" watchObservedRunningTime="2026-11-03 09:58:17.483849893 +0000 UTC m=+4721.151890633" Nov 03 09:58:17.535118 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:17.535085 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:18.540530 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:18.540501 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:19.371579 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:19.371550 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:19.372574 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:19.372558 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:19.479356 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:19.479327 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:19.656281 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:19.656193 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:19.789955 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:19.789912 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:22.791300 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:22.791266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:22.972335 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:22.972302 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:25.696293 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.696251 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:58:25.700101 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.700084 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.806820 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.806791 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a3965d89-serving-cert\"" Nov 03 09:58:25.808702 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.808678 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:58:25.862807 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.862777 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.863010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.862846 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.863010 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.862985 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rsz5\" (UniqueName: \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.863120 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.863058 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.888303 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.888273 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:25.964073 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.963992 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.964073 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.964062 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.964344 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.964094 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rsz5\" (UniqueName: \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.964344 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.964166 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.964714 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.964693 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.966660 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.966640 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:25.966759 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:25.966678 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:26.023830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.023791 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rsz5\" (UniqueName: \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\") pod \"maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:26.050600 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.050570 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:26.174755 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.174719 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:26.314316 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.314223 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:26.686878 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.686853 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:58:26.688882 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:58:26.688853 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da5635f_89f6_4052_9500_607a9bb510b6.slice/crio-beb591e9f9b27c71c037584df5465839452f0060de87362bc3687500836f4f7b WatchSource:0}: Error finding container beb591e9f9b27c71c037584df5465839452f0060de87362bc3687500836f4f7b: Status 404 returned error can't find the container with id beb591e9f9b27c71c037584df5465839452f0060de87362bc3687500836f4f7b Nov 03 09:58:26.690301 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:26.690282 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 09:58:27.412255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:27.412219 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" event={"ID":"1da5635f-89f6-4052-9500-607a9bb510b6","Type":"ContainerStarted","Data":"beb591e9f9b27c71c037584df5465839452f0060de87362bc3687500836f4f7b"} Nov 03 09:58:27.533696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:27.533660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:28.419960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:28.419923 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" event={"ID":"1da5635f-89f6-4052-9500-607a9bb510b6","Type":"ContainerStarted","Data":"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb"} Nov 03 09:58:28.420408 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:28.420047 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:28.538707 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:28.538645 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" podStartSLOduration=2.706993265 podStartE2EDuration="3.538631346s" totalImagesPullingTime="831.638081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:58:28.536864006 +0000 UTC m=+4732.204904744" watchObservedRunningTime="2026-11-03 09:58:28.538631346 +0000 UTC m=+4732.206672085" Nov 03 09:58:29.931327 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:29.931277 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:30.069786 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:30.069748 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:31.031035 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:31.031001 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:34.431685 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:34.431656 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:58:35.561800 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:58:35.561765 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:58:36.039409 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:36.039314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:36.165729 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:36.165682 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:41.031521 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:41.031484 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:43.495414 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.495379 4543 generic.go:356] "Generic (PLEG): container finished" podID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerID="1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f" exitCode=0 Nov 03 09:58:43.495857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.495436 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerDied","Data":"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f"} Nov 03 09:58:43.495857 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.495815 4543 scope.go:122] "RemoveContainer" containerID="1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f" Nov 03 09:58:43.497096 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.497065 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:43.497188 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.497165 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:43.497231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:43.497198 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:58:44.507074 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:44.507037 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerStarted","Data":"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c"} Nov 03 09:58:44.535999 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:44.535971 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:44.536343 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:44.536321 4543 patch_prober.go:29] interesting pod/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 09:58:44.536411 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:44.536367 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 09:58:44.604830 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:44.604768 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:45.535443 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:45.535403 4543 patch_prober.go:29] interesting pod/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 09:58:45.535938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:45.535459 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 09:58:46.536032 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:46.535996 4543 patch_prober.go:29] interesting pod/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 09:58:46.536573 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:46.536066 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 09:58:46.615476 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:46.615441 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:47.535698 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:47.535658 4543 patch_prober.go:29] interesting pod/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 09:58:47.535919 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:47.535721 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 09:58:48.535816 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:48.535783 4543 patch_prober.go:29] interesting pod/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 03 09:58:48.536255 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:48.535833 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 03 09:58:49.371923 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:49.371874 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:49.539746 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:49.539715 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:49.540812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:49.540793 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:58:51.929307 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:51.929249 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:58:56.938867 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:58:56.938836 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:01.971080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:01.971029 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:05.561077 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:59:05.561040 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:59:06.941606 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:06.941571 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:07.895093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:07.895058 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:08.024234 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:08.024194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:11.469783 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:11.469748 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:11.842632 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:11.842535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:15.353066 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:15.353020 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:16.216975 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.216940 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:59:16.217196 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.217174 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" podUID="1da5635f-89f6-4052-9500-607a9bb510b6" containerName="maas-api" containerID="cri-o://a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb" gracePeriod=30 Nov 03 09:59:16.310486 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.310398 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:16.459658 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.459630 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:59:16.579836 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.579761 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\" (UniqueName: \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\") pod \"1da5635f-89f6-4052-9500-607a9bb510b6\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " Nov 03 09:59:16.579985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.579859 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\") pod \"1da5635f-89f6-4052-9500-607a9bb510b6\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " Nov 03 09:59:16.579985 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.579919 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\") pod \"1da5635f-89f6-4052-9500-607a9bb510b6\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " Nov 03 09:59:16.580199 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.580033 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\") pod \"1da5635f-89f6-4052-9500-607a9bb510b6\" (UID: \"1da5635f-89f6-4052-9500-607a9bb510b6\") " Nov 03 09:59:16.580726 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.580524 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca" pod "1da5635f-89f6-4052-9500-607a9bb510b6" (UID: "1da5635f-89f6-4052-9500-607a9bb510b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:59:16.583178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.582690 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5" pod "1da5635f-89f6-4052-9500-607a9bb510b6" (UID: "1da5635f-89f6-4052-9500-607a9bb510b6"). InnerVolumeSpecName "kube-api-access-5rsz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:16.583178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.582702 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs" pod "1da5635f-89f6-4052-9500-607a9bb510b6" (UID: "1da5635f-89f6-4052-9500-607a9bb510b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:59:16.583178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.582804 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls" pod "1da5635f-89f6-4052-9500-607a9bb510b6" (UID: "1da5635f-89f6-4052-9500-607a9bb510b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 09:59:16.670277 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.670243 4543 generic.go:356] "Generic (PLEG): container finished" podID="1da5635f-89f6-4052-9500-607a9bb510b6" containerID="a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb" exitCode=0 Nov 03 09:59:16.670474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.670288 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" Nov 03 09:59:16.670474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.670342 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" event={"ID":"1da5635f-89f6-4052-9500-607a9bb510b6","Type":"ContainerDied","Data":"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb"} Nov 03 09:59:16.670474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.670397 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8" event={"ID":"1da5635f-89f6-4052-9500-607a9bb510b6","Type":"ContainerDied","Data":"beb591e9f9b27c71c037584df5465839452f0060de87362bc3687500836f4f7b"} Nov 03 09:59:16.670474 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.670422 4543 scope.go:122] "RemoveContainer" containerID="a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb" Nov 03 09:59:16.680020 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680002 4543 scope.go:122] "RemoveContainer" containerID="a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb" Nov 03 09:59:16.680335 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:59:16.680316 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb\": container with ID starting with a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb not found: ID does not exist" containerID="a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb" Nov 03 09:59:16.680417 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680348 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb"} err="failed to get container status \"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb\": rpc error: code = NotFound desc = could not find container \"a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb\": container with ID starting with a7068647eb42c75baa7b626da4e5411cb0f47c09ea95951e13d58126c4c3a9bb not found: ID does not exist" Nov 03 09:59:16.680776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680760 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rsz5\" (UniqueName: \"kubernetes.io/projected/1da5635f-89f6-4052-9500-607a9bb510b6-kube-api-access-5rsz5\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:16.680815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680780 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:16.680815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680793 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da5635f-89f6-4052-9500-607a9bb510b6-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:16.680815 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.680803 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da5635f-89f6-4052-9500-607a9bb510b6-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:16.838648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.838564 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:59:16.846510 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:16.846483 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a3965d89-5995f4d87-wwdk8"] Nov 03 09:59:17.849972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:17.849935 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:18.136178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:18.136129 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:18.449945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:18.449868 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da5635f-89f6-4052-9500-607a9bb510b6" path="/var/lib/kubelet/pods/1da5635f-89f6-4052-9500-607a9bb510b6/volumes" Nov 03 09:59:22.226116 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:22.226070 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:22.230688 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:22.230658 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:59:22.230939 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:22.230916 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" containerID="cri-o://2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c" gracePeriod=30 Nov 03 09:59:27.486038 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.486014 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:59:27.576522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576433 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576522 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576484 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576546 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576584 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576666 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576718 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.576768 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576751 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.577044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576866 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:59:27.577044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576875 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:59:27.577044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576947 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.577044 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576998 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\") pod \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\" (UID: \"06416e7c-2d65-4444-bd96-bcfbf54beb4d\") " Nov 03 09:59:27.577308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.576996 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:59:27.577308 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577200 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:59:27.577776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577492 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.577776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577517 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.577776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577532 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.577776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577546 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.577776 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.577674 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 09:59:27.579055 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.579036 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 09:59:27.579174 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.579111 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "kube-api-access-d5xtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:27.579399 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.579384 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 09:59:27.579462 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.579399 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token" pod "06416e7c-2d65-4444-bd96-bcfbf54beb4d" (UID: "06416e7c-2d65-4444-bd96-bcfbf54beb4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 09:59:27.678183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.678120 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5xtq\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-kube-api-access-d5xtq\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.678183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.678172 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.678183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.678183 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.678183 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.678191 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.678459 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.678200 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06416e7c-2d65-4444-bd96-bcfbf54beb4d-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 09:59:27.725563 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.725534 4543 generic.go:356] "Generic (PLEG): container finished" podID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerID="2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c" exitCode=0 Nov 03 09:59:27.725751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.725582 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerDied","Data":"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c"} Nov 03 09:59:27.725751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.725601 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" Nov 03 09:59:27.725751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.725649 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg" event={"ID":"06416e7c-2d65-4444-bd96-bcfbf54beb4d","Type":"ContainerDied","Data":"827d970415cc6db82f1c939795b2b8d7150b5955163621232dc4e9cadf0b4998"} Nov 03 09:59:27.725751 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.725672 4543 scope.go:122] "RemoveContainer" containerID="2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c" Nov 03 09:59:27.737648 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.737623 4543 scope.go:122] "RemoveContainer" containerID="1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f" Nov 03 09:59:27.746684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.746664 4543 scope.go:122] "RemoveContainer" containerID="2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c" Nov 03 09:59:27.746965 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:59:27.746942 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c\": container with ID starting with 2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c not found: ID does not exist" containerID="2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c" Nov 03 09:59:27.747079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.746971 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c"} err="failed to get container status \"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c\": rpc error: code = NotFound desc = could not find container \"2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c\": container with ID starting with 2284372a14896d1b84fc0c65c6fea8bf07aba6b60d37dd12731188aa18bac04c not found: ID does not exist" Nov 03 09:59:27.747079 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.746990 4543 scope.go:122] "RemoveContainer" containerID="1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f" Nov 03 09:59:27.747243 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:59:27.747222 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f\": container with ID starting with 1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f not found: ID does not exist" containerID="1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f" Nov 03 09:59:27.747292 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.747250 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f"} err="failed to get container status \"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f\": rpc error: code = NotFound desc = could not find container \"1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f\": container with ID starting with 1f379bf261057d241e9a919abe0664fa2cf3787409b75821def538b5a48eb64f not found: ID does not exist" Nov 03 09:59:27.865403 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.865374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 09:59:27.896149 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.896099 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:59:27.900185 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:27.900158 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a3965d89-openshift-default-7ddc478bf5-zjzcg"] Nov 03 09:59:28.450592 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.450562 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" path="/var/lib/kubelet/pods/06416e7c-2d65-4444-bd96-bcfbf54beb4d/volumes" Nov 03 09:59:28.734580 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.734490 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 09:59:28.735065 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735048 4543 cpu_manager.go:414] "containerMap: removing container" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:28.735175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735068 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:28.735175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735105 4543 cpu_manager.go:414] "containerMap: removing container" podUID="1da5635f-89f6-4052-9500-607a9bb510b6" containerName="maas-api" Nov 03 09:59:28.735175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735114 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da5635f-89f6-4052-9500-607a9bb510b6" containerName="maas-api" Nov 03 09:59:28.735353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735261 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da5635f-89f6-4052-9500-607a9bb510b6" containerName="maas-api" Nov 03 09:59:28.735353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735285 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:28.735353 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.735296 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:28.740086 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.740063 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.805278 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.805247 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 09:59:28.852686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.852652 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 09:59:28.852686 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.852678 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k54gs\"" Nov 03 09:59:28.852913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.852709 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 09:59:28.852913 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.852724 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 09:59:28.888535 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.888502 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.888732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.888554 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.888732 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.888676 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzjsb\" (UniqueName: \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.989835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.989738 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.989835 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.989788 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.990093 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.989839 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzjsb\" (UniqueName: \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.992441 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.992417 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:28.992557 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:28.992470 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:29.049423 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:29.049392 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzjsb\" (UniqueName: \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\") pod \"maas-controller-7dd5f74f7b-65nwt\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:29.053080 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:29.053058 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:29.427879 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:29.427851 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 09:59:29.429572 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:59:29.429549 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec74622e_b3c6_4029_9041_0122d46a96ba.slice/crio-465a879c7209cbc3ba4d384a6f53d0f9816074d0c2e62451fcadc694e872aef7 WatchSource:0}: Error finding container 465a879c7209cbc3ba4d384a6f53d0f9816074d0c2e62451fcadc694e872aef7: Status 404 returned error can't find the container with id 465a879c7209cbc3ba4d384a6f53d0f9816074d0c2e62451fcadc694e872aef7 Nov 03 09:59:29.737905 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:29.737816 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" event={"ID":"ec74622e-b3c6-4029-9041-0122d46a96ba","Type":"ContainerStarted","Data":"465a879c7209cbc3ba4d384a6f53d0f9816074d0c2e62451fcadc694e872aef7"} Nov 03 09:59:30.744294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:30.744203 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" event={"ID":"ec74622e-b3c6-4029-9041-0122d46a96ba","Type":"ContainerStarted","Data":"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876"} Nov 03 09:59:30.744294 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:30.744281 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:30.861938 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:30.861885 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" podStartSLOduration=1.904231649 podStartE2EDuration="2.861871519s" totalImagesPullingTime="957.63987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:59:30.861762941 +0000 UTC m=+4794.529803679" watchObservedRunningTime="2026-11-03 09:59:30.861871519 +0000 UTC m=+4794.529912258" Nov 03 09:59:35.561032 ip-10-0-131-117 kubenswrapper[4543]: E1103 09:59:35.560996 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 09:59:37.145326 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:37.145304 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:59:37.195863 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:37.195838 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 09:59:41.751889 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:41.751859 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 09:59:46.705231 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.705196 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 09:59:46.705645 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.705585 4543 cpu_manager.go:414] "containerMap: removing container" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:46.705645 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.705595 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06416e7c-2d65-4444-bd96-bcfbf54beb4d" containerName="istio-proxy" Nov 03 09:59:46.709926 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.709909 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.755797 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.755766 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 09:59:46.759319 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.759297 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.764812 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.764788 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 09:59:46.764947 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.764887 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\"" Nov 03 09:59:46.809467 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.809433 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 09:59:46.819605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.819523 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 09:59:46.859605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859565 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.859605 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859606 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.859819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859622 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.859819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859732 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.859819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859779 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.859819 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859807 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.859960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859871 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.859960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859904 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.859960 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859944 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.860057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.859986 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.860057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860002 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.860057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860028 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bp5\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.860057 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860044 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.860225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860120 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.860225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860152 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.860225 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860206 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5s4z\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.860324 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860258 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.860324 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.860297 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961403 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961447 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961465 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961496 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961491 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961575 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bp5\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961593 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961627 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961643 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961661 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5s4z\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961678 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961718 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961749 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961774 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961795 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961822 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961838 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.961865 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961870 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961878 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961898 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961905 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.961949 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962171 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962180 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962338 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962432 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.962681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962636 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.963370 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.962955 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.964684 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.964658 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:46.964805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.964713 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.964805 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.964742 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:46.964940 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:46.964922 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:47.021332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.021298 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:47.021332 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.021324 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:47.021534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.021361 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bp5\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7s6wh\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:47.021534 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.021420 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5s4z\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gtk7l\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:47.022177 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.022161 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:47.071711 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.071628 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:47.450209 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.450179 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 09:59:47.451875 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:59:47.451847 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1907c83_2863_4e88_8995_356912ee98a2.slice/crio-cab1923dbb23f4acfef1ed1e5bf8a0291b5438818d69a8070bb6526e6d7cbdcb WatchSource:0}: Error finding container cab1923dbb23f4acfef1ed1e5bf8a0291b5438818d69a8070bb6526e6d7cbdcb: Status 404 returned error can't find the container with id cab1923dbb23f4acfef1ed1e5bf8a0291b5438818d69a8070bb6526e6d7cbdcb Nov 03 09:59:47.454133 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.454104 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.454247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.454196 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.454247 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.454241 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.503478 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.503452 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 09:59:47.506318 ip-10-0-131-117 kubenswrapper[4543]: W1103 09:59:47.506290 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13546769_d45e_45c9_be20_621f4f329f79.slice/crio-013e6556959972bdab4cea76cf35e59b1ba5d518bc2b7f3691c596930a36ba2a WatchSource:0}: Error finding container 013e6556959972bdab4cea76cf35e59b1ba5d518bc2b7f3691c596930a36ba2a: Status 404 returned error can't find the container with id 013e6556959972bdab4cea76cf35e59b1ba5d518bc2b7f3691c596930a36ba2a Nov 03 09:59:47.508818 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.508785 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.508894 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.508878 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.508932 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.508915 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 09:59:47.828597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.828504 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" event={"ID":"13546769-d45e-45c9-be20-621f4f329f79","Type":"ContainerStarted","Data":"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d"} Nov 03 09:59:47.828597 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.828563 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" event={"ID":"13546769-d45e-45c9-be20-621f4f329f79","Type":"ContainerStarted","Data":"013e6556959972bdab4cea76cf35e59b1ba5d518bc2b7f3691c596930a36ba2a"} Nov 03 09:59:47.830625 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.830593 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" event={"ID":"d1907c83-2863-4e88-8995-356912ee98a2","Type":"ContainerStarted","Data":"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c"} Nov 03 09:59:47.830763 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.830650 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" event={"ID":"d1907c83-2863-4e88-8995-356912ee98a2","Type":"ContainerStarted","Data":"cab1923dbb23f4acfef1ed1e5bf8a0291b5438818d69a8070bb6526e6d7cbdcb"} Nov 03 09:59:47.958008 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:47.957954 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podStartSLOduration=1.957938622 podStartE2EDuration="1.957938622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:59:47.95573458 +0000 UTC m=+4811.623775320" watchObservedRunningTime="2026-11-03 09:59:47.957938622 +0000 UTC m=+4811.625979362" Nov 03 09:59:48.025175 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.025115 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:48.030166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.029000 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:48.030166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.029063 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:48.071928 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.071866 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:48.073090 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.073067 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:48.073224 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.073118 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:48.078937 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:48.078846 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podStartSLOduration=2.078834153 podStartE2EDuration="2.078834153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:59:48.077113053 +0000 UTC m=+4811.745153805" watchObservedRunningTime="2026-11-03 09:59:48.078834153 +0000 UTC m=+4811.746874891" Nov 03 09:59:49.024166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:49.023653 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:49.024166 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:49.023729 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:49.072129 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:49.072091 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:49.072321 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:49.072163 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:50.028167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:50.027851 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:50.028167 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:50.027914 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:50.072762 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:50.072724 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:50.072945 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:50.072783 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:51.024171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:51.023659 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:51.024171 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:51.023724 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:51.072567 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:51.072522 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:51.072972 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:51.072587 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:52.032289 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:52.032253 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:52.033336 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:52.033303 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:52.072197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:52.072090 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:52.072197 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:52.072152 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:53.028169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:53.027717 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:53.028169 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:53.027795 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:53.080407 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:53.080368 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:53.080589 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:53.080420 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:54.023612 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:54.023575 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:54.023891 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:54.023644 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:54.072355 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:54.072321 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:54.072795 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:54.072387 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:55.022490 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:55.022452 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 09:59:55.022681 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:55.022506 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 09:59:55.073027 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:55.072995 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 03 09:59:55.073603 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:55.073046 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 03 09:59:56.029202 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.029168 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:56.030246 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.030201 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:56.030937 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.030912 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 09:59:56.078696 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.078644 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:56.079178 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.078959 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 09:59:56.079886 ip-10-0-131-117 kubenswrapper[4543]: I1103 09:59:56.079864 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 10:00:03.912946 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:03.912910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:00:04.037412 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.037377 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 10:00:04.037685 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.037659 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" podUID="ec74622e-b3c6-4029-9041-0122d46a96ba" containerName="manager" containerID="cri-o://6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876" gracePeriod=30 Nov 03 10:00:04.290034 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.290009 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 10:00:04.373589 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.373554 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\") pod \"ec74622e-b3c6-4029-9041-0122d46a96ba\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " Nov 03 10:00:04.373766 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.373608 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\") pod \"ec74622e-b3c6-4029-9041-0122d46a96ba\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " Nov 03 10:00:04.373766 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.373653 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\" (UniqueName: \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\") pod \"ec74622e-b3c6-4029-9041-0122d46a96ba\" (UID: \"ec74622e-b3c6-4029-9041-0122d46a96ba\") " Nov 03 10:00:04.376041 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.376014 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb" pod "ec74622e-b3c6-4029-9041-0122d46a96ba" (UID: "ec74622e-b3c6-4029-9041-0122d46a96ba"). InnerVolumeSpecName "kube-api-access-mzjsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:00:04.376193 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.376010 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs" pod "ec74622e-b3c6-4029-9041-0122d46a96ba" (UID: "ec74622e-b3c6-4029-9041-0122d46a96ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:00:04.376193 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.376020 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert" pod "ec74622e-b3c6-4029-9041-0122d46a96ba" (UID: "ec74622e-b3c6-4029-9041-0122d46a96ba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:00:04.474797 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.474768 4543 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-webhook-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:04.474797 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.474792 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec74622e-b3c6-4029-9041-0122d46a96ba-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:04.474797 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.474803 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzjsb\" (UniqueName: \"kubernetes.io/projected/ec74622e-b3c6-4029-9041-0122d46a96ba-kube-api-access-mzjsb\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:00:04.942897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.942853 4543 generic.go:356] "Generic (PLEG): container finished" podID="ec74622e-b3c6-4029-9041-0122d46a96ba" containerID="6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876" exitCode=0 Nov 03 10:00:04.943364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.942906 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" Nov 03 10:00:04.943364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.942939 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" event={"ID":"ec74622e-b3c6-4029-9041-0122d46a96ba","Type":"ContainerDied","Data":"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876"} Nov 03 10:00:04.943364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.942998 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-65nwt" event={"ID":"ec74622e-b3c6-4029-9041-0122d46a96ba","Type":"ContainerDied","Data":"465a879c7209cbc3ba4d384a6f53d0f9816074d0c2e62451fcadc694e872aef7"} Nov 03 10:00:04.943364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.943022 4543 scope.go:122] "RemoveContainer" containerID="6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876" Nov 03 10:00:04.951837 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.951814 4543 scope.go:122] "RemoveContainer" containerID="6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876" Nov 03 10:00:04.952113 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:00:04.952092 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876\": container with ID starting with 6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876 not found: ID does not exist" containerID="6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876" Nov 03 10:00:04.952219 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:04.952120 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876"} err="failed to get container status \"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876\": rpc error: code = NotFound desc = could not find container \"6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876\": container with ID starting with 6b044cc08d8353bc09d1f6f00d5bea624255ad4d4d9c2c999514a4ba6626d876 not found: ID does not exist" Nov 03 10:00:05.113936 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:05.113901 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 10:00:05.164592 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:05.164557 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-65nwt"] Nov 03 10:00:05.560745 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:00:05.560710 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:00:06.353548 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.353516 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:00:06.450774 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.450730 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec74622e-b3c6-4029-9041-0122d46a96ba" path="/var/lib/kubelet/pods/ec74622e-b3c6-4029-9041-0122d46a96ba/volumes" Nov 03 10:00:06.497614 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.497580 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-kfmgw"] Nov 03 10:00:06.498004 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.497991 4543 cpu_manager.go:414] "containerMap: removing container" podUID="ec74622e-b3c6-4029-9041-0122d46a96ba" containerName="manager" Nov 03 10:00:06.498045 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.498005 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec74622e-b3c6-4029-9041-0122d46a96ba" containerName="manager" Nov 03 10:00:06.498085 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.498078 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec74622e-b3c6-4029-9041-0122d46a96ba" containerName="manager" Nov 03 10:00:06.502560 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.502543 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.587967 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.587929 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 10:00:06.594428 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.594398 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-webhook-cert\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.594588 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.594431 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-metrics-certs\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.594588 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.594458 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqvzj\" (UniqueName: \"kubernetes.io/projected/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-kube-api-access-pqvzj\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.606790 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.606711 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 10:00:06.606948 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.606837 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k54gs\"" Nov 03 10:00:06.607016 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.606952 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 10:00:06.609183 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.609163 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-kfmgw"] Nov 03 10:00:06.695674 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.695638 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-webhook-cert\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.695674 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.695678 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-metrics-certs\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.695897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.695718 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqvzj\" (UniqueName: \"kubernetes.io/projected/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-kube-api-access-pqvzj\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.698383 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.698352 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-webhook-cert\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.698506 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.698411 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-metrics-certs\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.755048 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.755019 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqvzj\" (UniqueName: \"kubernetes.io/projected/3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f-kube-api-access-pqvzj\") pod \"maas-controller-7dd5f74f7b-kfmgw\" (UID: \"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f\") " pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:06.816228 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:06.816189 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:07.189862 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:07.189837 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-kfmgw"] Nov 03 10:00:07.191812 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:00:07.191786 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4062a6_2fa5_49e0_ae6a_9f30f42d3e7f.slice/crio-e15d348ff66a8ca9774a4eb793b3914fb2c2a559ea6099a7df106a77971786d2 WatchSource:0}: Error finding container e15d348ff66a8ca9774a4eb793b3914fb2c2a559ea6099a7df106a77971786d2: Status 404 returned error can't find the container with id e15d348ff66a8ca9774a4eb793b3914fb2c2a559ea6099a7df106a77971786d2 Nov 03 10:00:07.959673 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:07.959639 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" event={"ID":"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f","Type":"ContainerStarted","Data":"e15d348ff66a8ca9774a4eb793b3914fb2c2a559ea6099a7df106a77971786d2"} Nov 03 10:00:08.965952 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:08.965905 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" event={"ID":"3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f","Type":"ContainerStarted","Data":"5f680d421dd943bd7fc0b752d5f21c9988bec5a1bc30633be7161b68b63165ac"} Nov 03 10:00:08.966421 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:08.966102 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:09.083960 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:09.083909 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" podStartSLOduration=2.316666984 podStartE2EDuration="3.0838962s" totalImagesPullingTime="767.229216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:00:09.082941376 +0000 UTC m=+4832.750982130" watchObservedRunningTime="2026-11-03 10:00:09.0838962 +0000 UTC m=+4832.751936938" Nov 03 10:00:19.973199 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:19.973165 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dd5f74f7b-kfmgw" Nov 03 10:00:28.270580 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:28.270541 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:00:35.561527 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:00:35.561472 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:00:57.073367 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:57.073300 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:00:57.074088 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:57.074053 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:00:57.120332 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:57.120288 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:00:57.120551 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:00:57.120376 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:02.315898 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.315859 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:01:02.319452 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.319435 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.425583 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.425549 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:01:02.440109 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440077 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440277 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440118 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440277 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440249 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440357 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440282 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440357 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440300 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440430 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440352 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440499 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440485 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440539 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440509 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88b4\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.440574 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.440536 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541374 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541340 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541374 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541374 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541413 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541459 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541476 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88b4\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541502 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541524 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.541626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541606 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.542069 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541801 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.542069 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.541828 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.542069 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.542017 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.542231 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.542163 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.542274 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.542234 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.543999 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.543975 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.544227 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.544210 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.600474 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.600405 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.600650 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.600585 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88b4\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9tmmz\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:02.633387 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:02.633364 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:03.059424 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.059399 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:01:03.061004 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:01:03.060979 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec7421a_e681_494c_989c_e780b975158f.slice/crio-ee1b0908435bf6c5fa93c0074cfde3e77865cb467011964f3ecf382551c2821a WatchSource:0}: Error finding container ee1b0908435bf6c5fa93c0074cfde3e77865cb467011964f3ecf382551c2821a: Status 404 returned error can't find the container with id ee1b0908435bf6c5fa93c0074cfde3e77865cb467011964f3ecf382551c2821a Nov 03 10:01:03.063345 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.063313 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:03.063421 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.063380 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:03.063421 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.063411 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:03.249905 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.249865 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" event={"ID":"bec7421a-e681-494c-989c-e780b975158f","Type":"ContainerStarted","Data":"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243"} Nov 03 10:01:03.250092 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.249933 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" event={"ID":"bec7421a-e681-494c-989c-e780b975158f","Type":"ContainerStarted","Data":"ee1b0908435bf6c5fa93c0074cfde3e77865cb467011964f3ecf382551c2821a"} Nov 03 10:01:03.372861 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.372795 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podStartSLOduration=1.37277927 podStartE2EDuration="1.37277927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:03.370577098 +0000 UTC m=+4887.038617837" watchObservedRunningTime="2026-11-03 10:01:03.37277927 +0000 UTC m=+4887.040820025" Nov 03 10:01:03.633541 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:03.633451 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:04.635662 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:04.635623 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:04.636217 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:04.635674 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:05.578625 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:01:05.578586 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:01:05.637123 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:05.636628 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:05.637123 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:05.636710 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:06.636099 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:06.635622 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:06.636099 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:06.635687 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:07.634332 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:07.634285 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:07.634836 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:07.634352 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:08.633946 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:08.633899 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:08.634209 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:08.633958 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:09.634454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:09.634412 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:09.634861 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:09.634494 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:10.634329 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:10.634286 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 10:01:10.634524 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:10.634373 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 10:01:12.675322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:12.675282 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:12.675790 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:12.675356 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:13.615162 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:13.615110 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:13.615769 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:13.615743 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:13.616399 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:13.616378 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:01:16.830536 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:16.830447 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:01:17.422558 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.422521 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:01:17.427035 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.427012 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.533183 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.533123 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:01:17.611981 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.611945 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612206 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.611988 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612206 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612093 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612206 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612193 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612346 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612280 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612346 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612323 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612450 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612354 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9md\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612450 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612382 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.612531 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.612438 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713585 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713496 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713585 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713547 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713585 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713572 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713612 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9md\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713637 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713654 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713784 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.713864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713833 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714233 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.713892 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714233 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.714055 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714233 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.714126 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714233 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.714200 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714434 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.714242 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.714879 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.714855 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.716367 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.716341 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.716472 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.716374 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.774576 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.774539 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:17.774951 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:17.774691 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9md\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-h9rgl\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:18.042910 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:18.042828 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:18.469285 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:18.469261 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:01:18.470893 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:01:18.470869 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37aaef4_8a1f_4f2d_a97e_558328072820.slice/crio-a11bd6c1823e79f215d9e4386dfa2bbcc70b6e9aa21359578fca28a9bb98bd88 WatchSource:0}: Error finding container a11bd6c1823e79f215d9e4386dfa2bbcc70b6e9aa21359578fca28a9bb98bd88: Status 404 returned error can't find the container with id a11bd6c1823e79f215d9e4386dfa2bbcc70b6e9aa21359578fca28a9bb98bd88 Nov 03 10:01:18.473087 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:18.473056 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:18.473182 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:18.473126 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:18.473182 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:18.473177 4543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 10:01:19.336524 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:19.336483 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" event={"ID":"a37aaef4-8a1f-4f2d-a97e-558328072820","Type":"ContainerStarted","Data":"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d"} Nov 03 10:01:19.337000 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:19.336582 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" event={"ID":"a37aaef4-8a1f-4f2d-a97e-558328072820","Type":"ContainerStarted","Data":"a11bd6c1823e79f215d9e4386dfa2bbcc70b6e9aa21359578fca28a9bb98bd88"} Nov 03 10:01:19.510738 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:19.510675 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podStartSLOduration=2.5106619180000003 podStartE2EDuration="2.510661918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:19.508913545 +0000 UTC m=+4903.176954289" watchObservedRunningTime="2026-11-03 10:01:19.510661918 +0000 UTC m=+4903.178702656" Nov 03 10:01:20.044008 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:20.043604 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:20.045470 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:20.045436 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:20.045608 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:20.045506 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:21.044218 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:21.044177 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:21.044790 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:21.044256 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:22.043356 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:22.043320 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:22.043551 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:22.043385 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:23.043490 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:23.043455 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:23.043907 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:23.043552 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:24.044032 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:24.044003 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:24.044543 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:24.044052 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:25.043314 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:25.043275 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:25.043544 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:25.043342 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:26.044293 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:26.044254 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 10:01:26.044740 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:26.044325 4543 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 10:01:27.047904 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:27.047874 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:27.048346 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:27.048311 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:27.049199 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:27.049177 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:01:28.561420 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:28.561382 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:01:35.561845 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:01:35.561803 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:01:50.006619 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:50.006584 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:01:58.344754 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:58.344717 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:01:59.080370 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.079586 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:01:59.094469 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.094439 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.137526 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.137488 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:01:59.163537 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.163487 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:01:59.181081 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.181032 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.181307 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.181090 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfvl\" (UniqueName: \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.183522 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.183494 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8906966c-epp-sa-dockercfg-5vw5m\"" Nov 03 10:01:59.183941 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.183893 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-8906966c-kserve-self-signed-certs\"" Nov 03 10:01:59.282742 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.282702 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.283026 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.283003 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfvl\" (UniqueName: \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.300976 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.300914 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.349056 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.348974 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfvl\" (UniqueName: \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\") pod \"e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.423060 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.423021 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:01:59.694190 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.691320 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:01:59.694190 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.691387 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:01:59.832434 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:01:59.831976 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:01:59.834386 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:01:59.834352 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952358a2_0923_4b68_bc62_33276c5c06c7.slice/crio-8974e8648d24c2d8b797899bdee10dd52fd0446d412d4408f41caf339cbed5de WatchSource:0}: Error finding container 8974e8648d24c2d8b797899bdee10dd52fd0446d412d4408f41caf339cbed5de: Status 404 returned error can't find the container with id 8974e8648d24c2d8b797899bdee10dd52fd0446d412d4408f41caf339cbed5de Nov 03 10:02:00.694839 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:00.694804 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" event={"ID":"952358a2-0923-4b68-bc62-33276c5c06c7","Type":"ContainerStarted","Data":"8974e8648d24c2d8b797899bdee10dd52fd0446d412d4408f41caf339cbed5de"} Nov 03 10:02:02.706449 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:02.706410 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" event={"ID":"952358a2-0923-4b68-bc62-33276c5c06c7","Type":"ContainerStarted","Data":"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639"} Nov 03 10:02:02.706898 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:02.706498 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:02:02.823054 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:02.823003 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podStartSLOduration=1.9686301880000001 podStartE2EDuration="3.822988607s" totalImagesPullingTime="1.854358419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:02:02.821874969 +0000 UTC m=+4946.489915706" watchObservedRunningTime="2026-11-03 10:02:02.822988607 +0000 UTC m=+4946.491029346" Nov 03 10:02:03.441925 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:03.441887 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:02:05.653640 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:02:05.653557 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:02:17.180798 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:17.180758 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:02:21.064868 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:21.064833 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:02:21.195353 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:21.195289 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:02:33.712711 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:33.712681 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:02:35.560597 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:02:35.560558 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:02:38.901410 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:02:38.901374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:05.559747 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:03:05.559658 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:03:18.609062 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:18.609026 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:26.765180 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:26.765132 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:03:26.771738 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:26.771698 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:26.912254 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:26.912219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:33.728165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:33.726231 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:03:35.560158 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:03:35.560106 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:03:39.244575 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:39.244535 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:41.786080 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:41.786036 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" containerID="cri-o://702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639" gracePeriod=45 Nov 03 10:03:43.713325 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:43.713288 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:03:44.839937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:44.839900 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:45.491708 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:45.491676 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:45.701564 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:45.701521 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:49.744391 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:49.744358 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:50.709282 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:50.709245 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:53.056330 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:53.056274 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:53.232131 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:53.232101 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:53.712821 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:53.712789 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:03:53.713000 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:53.712903 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:03:56.626082 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:56.626044 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:56.790652 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:56.790618 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:03:56.918351 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:03:56.918272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:01.812877 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:01.812839 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:03.712772 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:03.712730 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 10:04:05.560548 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:04:05.560512 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:04:06.799396 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:06.799352 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:11.792013 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:11.791981 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:12.043488 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.043430 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:04:12.141512 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.141472 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\" (UniqueName: \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\") pod \"952358a2-0923-4b68-bc62-33276c5c06c7\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " Nov 03 10:04:12.141740 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.141558 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\") pod \"952358a2-0923-4b68-bc62-33276c5c06c7\" (UID: \"952358a2-0923-4b68-bc62-33276c5c06c7\") " Nov 03 10:04:12.144714 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.144676 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs" pod "952358a2-0923-4b68-bc62-33276c5c06c7" (UID: "952358a2-0923-4b68-bc62-33276c5c06c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:04:12.145411 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.145372 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl" pod "952358a2-0923-4b68-bc62-33276c5c06c7" (UID: "952358a2-0923-4b68-bc62-33276c5c06c7"). InnerVolumeSpecName "kube-api-access-jkfvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:04:12.242499 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.242463 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkfvl\" (UniqueName: \"kubernetes.io/projected/952358a2-0923-4b68-bc62-33276c5c06c7-kube-api-access-jkfvl\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:04:12.242499 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.242493 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952358a2-0923-4b68-bc62-33276c5c06c7-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:04:12.476785 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.476756 4543 generic.go:356] "Generic (PLEG): container finished" podID="952358a2-0923-4b68-bc62-33276c5c06c7" containerID="702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639" exitCode=0 Nov 03 10:04:12.476935 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.476796 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" Nov 03 10:04:12.476935 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.476845 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" event={"ID":"952358a2-0923-4b68-bc62-33276c5c06c7","Type":"ContainerDied","Data":"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639"} Nov 03 10:04:12.476935 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.476895 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6" event={"ID":"952358a2-0923-4b68-bc62-33276c5c06c7","Type":"ContainerDied","Data":"8974e8648d24c2d8b797899bdee10dd52fd0446d412d4408f41caf339cbed5de"} Nov 03 10:04:12.476935 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.476918 4543 scope.go:122] "RemoveContainer" containerID="702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639" Nov 03 10:04:12.485330 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.485312 4543 scope.go:122] "RemoveContainer" containerID="702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639" Nov 03 10:04:12.485603 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:04:12.485585 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639\": container with ID starting with 702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639 not found: ID does not exist" containerID="702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639" Nov 03 10:04:12.485678 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.485611 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639"} err="failed to get container status \"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639\": rpc error: code = NotFound desc = could not find container \"702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639\": container with ID starting with 702106c6a3d5b420b36d932e854ec63c68ad2b5d5840b7c3c52c9f39febd8639 not found: ID does not exist" Nov 03 10:04:12.645811 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.645778 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:04:12.652641 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:12.652614 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-8906966c-kserve-router-scheduler-6c4c46bdd5-fnvc6"] Nov 03 10:04:14.451224 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:14.451192 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" path="/var/lib/kubelet/pods/952358a2-0923-4b68-bc62-33276c5c06c7/volumes" Nov 03 10:04:14.516198 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:14.516166 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:15.090815 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:15.090783 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:15.229987 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:15.229949 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:15.365840 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:15.365804 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:17.123357 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:17.123281 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:17.878258 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:17.878223 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:19.972115 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:19.972079 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:04:19.972623 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:19.972558 4543 cpu_manager.go:414] "containerMap: removing container" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" Nov 03 10:04:19.972623 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:19.972571 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" Nov 03 10:04:19.972749 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:19.972655 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="952358a2-0923-4b68-bc62-33276c5c06c7" containerName="main" Nov 03 10:04:19.977013 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:19.976993 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.034645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.034616 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:04:20.047120 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.047088 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fd27d2\"/\"openshift-service-ca.crt\"" Nov 03 10:04:20.053783 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.053755 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:20.082124 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.082097 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fd27d2\"/\"kube-root-ca.crt\"" Nov 03 10:04:20.082309 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.082110 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fd27d2\"/\"simulator-single-main-fd27d2-kserve-self-signed-certs\"" Nov 03 10:04:20.082309 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.082236 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fd27d2\"/\"default-dockercfg-chc6s\"" Nov 03 10:04:20.120261 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120205 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.120555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120268 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.120555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120313 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.120555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120387 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.120555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120422 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrp7\" (UniqueName: \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.120555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.120458 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221278 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221249 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221297 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrp7\" (UniqueName: \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221325 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221361 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221397 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221709 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221519 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221775 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221708 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221775 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221748 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.221854 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.221814 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.223826 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.223757 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.224081 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.224063 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.280171 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.280113 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrp7\" (UniqueName: \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\") pod \"simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.291032 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.291012 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:20.663291 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:20.663261 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:04:20.664790 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:04:20.664765 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d874f08_f0b3_44d1_92aa_fcb97117240f.slice/crio-bfb60683f31e7289d15a5056f930bac9aa5b3fbeaad476f21ed39aedaee9532e WatchSource:0}: Error finding container bfb60683f31e7289d15a5056f930bac9aa5b3fbeaad476f21ed39aedaee9532e: Status 404 returned error can't find the container with id bfb60683f31e7289d15a5056f930bac9aa5b3fbeaad476f21ed39aedaee9532e Nov 03 10:04:21.530563 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:21.530526 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" event={"ID":"8d874f08-f0b3-44d1-92aa-fcb97117240f","Type":"ContainerStarted","Data":"6d22325d3dc98e47b9479add2f4a3a29a24303cca66d1850f9a3f508d0ccfcf7"} Nov 03 10:04:21.530563 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:21.530569 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" event={"ID":"8d874f08-f0b3-44d1-92aa-fcb97117240f","Type":"ContainerStarted","Data":"bfb60683f31e7289d15a5056f930bac9aa5b3fbeaad476f21ed39aedaee9532e"} Nov 03 10:04:21.649795 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:21.649741 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" podStartSLOduration=2.649728135 podStartE2EDuration="2.649728135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:04:21.648401681 +0000 UTC m=+5085.316442423" watchObservedRunningTime="2026-11-03 10:04:21.649728135 +0000 UTC m=+5085.317768872" Nov 03 10:04:22.089437 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:22.089407 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:22.291957 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:22.291923 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:27.097616 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:27.097582 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:30.292077 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:30.292032 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:30.304233 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:30.304209 4543 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:30.586280 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:30.586198 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:04:32.020219 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:32.020187 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:35.560205 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:04:35.560166 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:04:36.996953 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:36.996913 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:37.177246 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:37.177221 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:04:37.229089 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:37.229062 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:04:39.582337 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:39.582303 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:41.984368 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:41.984332 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:46.993378 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:46.993343 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:52.001701 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:52.001665 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:04:56.997315 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:04:56.997280 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:01.985686 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:01.985651 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:05.564370 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:05:05.564319 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:05:06.996280 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:06.996250 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:11.983984 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:11.983950 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:17.000011 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:16.999922 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:18.219206 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:18.219172 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:22.005328 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:22.005292 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:27.005299 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:27.005256 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:29.879523 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:29.879486 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:30.348763 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:30.348671 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:31.986841 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:31.986794 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:34.770767 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:34.770727 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:34.799445 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:34.799415 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:05:34.928976 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:34.928939 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:35.560402 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:05:35.560369 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:05:35.964897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:35.964862 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:36.260432 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:36.260353 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:36.549986 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:36.549910 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:37.392605 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:37.392576 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:39.927403 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:39.927374 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:43.563120 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:43.563083 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:43.777920 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:43.777882 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:05:49.819987 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:49.819892 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" podUID="8d874f08-f0b3-44d1-92aa-fcb97117240f" containerName="main" containerID="cri-o://6d22325d3dc98e47b9479add2f4a3a29a24303cca66d1850f9a3f508d0ccfcf7" gracePeriod=45 Nov 03 10:05:49.976195 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:49.976133 4543 generic.go:356] "Generic (PLEG): container finished" podID="8d874f08-f0b3-44d1-92aa-fcb97117240f" containerID="6d22325d3dc98e47b9479add2f4a3a29a24303cca66d1850f9a3f508d0ccfcf7" exitCode=0 Nov 03 10:05:49.976362 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:49.976209 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" event={"ID":"8d874f08-f0b3-44d1-92aa-fcb97117240f","Type":"ContainerDied","Data":"6d22325d3dc98e47b9479add2f4a3a29a24303cca66d1850f9a3f508d0ccfcf7"} Nov 03 10:05:50.076383 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.076326 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:05:50.123112 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123079 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123121 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123268 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123409 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123352 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123409 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123396 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\" (UniqueName: \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123479 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123432 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\") pod \"8d874f08-f0b3-44d1-92aa-fcb97117240f\" (UID: \"8d874f08-f0b3-44d1-92aa-fcb97117240f\") " Nov 03 10:05:50.123513 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123467 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:05:50.123762 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123730 4543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-model-cache\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.123858 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123816 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:05:50.123935 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.123918 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:05:50.125749 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.125725 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:05:50.125875 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.125818 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:05:50.125967 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.125945 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7" pod "8d874f08-f0b3-44d1-92aa-fcb97117240f" (UID: "8d874f08-f0b3-44d1-92aa-fcb97117240f"). InnerVolumeSpecName "kube-api-access-pqrp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:05:50.224490 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.224448 4543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-tmp-dir\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.224490 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.224491 4543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d874f08-f0b3-44d1-92aa-fcb97117240f-tls-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.224720 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.224507 4543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-home\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.224720 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.224520 4543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d874f08-f0b3-44d1-92aa-fcb97117240f-dshm\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.224720 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.224534 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrp7\" (UniqueName: \"kubernetes.io/projected/8d874f08-f0b3-44d1-92aa-fcb97117240f-kube-api-access-pqrp7\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:05:50.982597 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.982567 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" Nov 03 10:05:50.982597 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.982588 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch" event={"ID":"8d874f08-f0b3-44d1-92aa-fcb97117240f","Type":"ContainerDied","Data":"bfb60683f31e7289d15a5056f930bac9aa5b3fbeaad476f21ed39aedaee9532e"} Nov 03 10:05:50.983220 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:50.982642 4543 scope.go:122] "RemoveContainer" containerID="6d22325d3dc98e47b9479add2f4a3a29a24303cca66d1850f9a3f508d0ccfcf7" Nov 03 10:05:51.152463 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:51.152425 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:05:51.202865 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:51.202829 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fd27d2/simulator-single-main-fd27d2-kserve-6ccbf856df-sk9ch"] Nov 03 10:05:52.450677 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:05:52.450641 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d874f08-f0b3-44d1-92aa-fcb97117240f" path="/var/lib/kubelet/pods/8d874f08-f0b3-44d1-92aa-fcb97117240f/volumes" Nov 03 10:06:05.561249 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:06:05.561213 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:06:10.586404 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:10.586371 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:10.725230 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:10.725185 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:12.503537 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.503506 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:12.720982 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.720944 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:12.721532 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.721513 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8d874f08-f0b3-44d1-92aa-fcb97117240f" containerName="main" Nov 03 10:06:12.721655 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.721534 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d874f08-f0b3-44d1-92aa-fcb97117240f" containerName="main" Nov 03 10:06:12.721655 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.721651 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d874f08-f0b3-44d1-92aa-fcb97117240f" containerName="main" Nov 03 10:06:12.726174 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.726152 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.830390 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.830304 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fd27d2-serving-cert\"" Nov 03 10:06:12.832963 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.832937 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:12.849396 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.849369 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccn2x\" (UniqueName: \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.849555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.849439 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.849612 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.849570 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.849612 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.849604 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.916655 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.916610 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:12.951183 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.951125 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.951362 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.951217 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.951362 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.951301 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccn2x\" (UniqueName: \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.951485 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.951418 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.951567 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:06:12.951550 4543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-serving-cert: secret "maas-api-e2e-worker-single-main-fd27d2-serving-cert" not found Nov 03 10:06:12.951641 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:06:12.951629 4543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls podName:b0899605-b3ca-4323-b44f-b60c475b64fc nodeName:}" failed. No retries permitted until 2026-11-03 10:06:13.451605103 +0000 UTC m=+5197.119645833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls") pod "maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" (UID: "b0899605-b3ca-4323-b44f-b60c475b64fc") : secret "maas-api-e2e-worker-single-main-fd27d2-serving-cert" not found Nov 03 10:06:12.951918 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.951897 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:12.953851 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:12.953822 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:13.010479 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.010443 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccn2x\" (UniqueName: \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:13.079276 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.079247 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:13.204547 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.204513 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:13.430625 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.430593 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:13.456680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.456594 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:13.459062 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.459036 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:13.638810 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:13.638781 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:14.013960 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:14.013925 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:14.015386 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:06:14.015360 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0899605_b3ca_4323_b44f_b60c475b64fc.slice/crio-dd05f98f43d2522bd1dc97470091a9e1ee7e2691c87404575c94889d8b3ebc27 WatchSource:0}: Error finding container dd05f98f43d2522bd1dc97470091a9e1ee7e2691c87404575c94889d8b3ebc27: Status 404 returned error can't find the container with id dd05f98f43d2522bd1dc97470091a9e1ee7e2691c87404575c94889d8b3ebc27 Nov 03 10:06:14.016916 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:14.016892 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:06:14.099049 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:14.099013 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" event={"ID":"b0899605-b3ca-4323-b44f-b60c475b64fc","Type":"ContainerStarted","Data":"dd05f98f43d2522bd1dc97470091a9e1ee7e2691c87404575c94889d8b3ebc27"} Nov 03 10:06:14.646119 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:14.646084 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:15.105463 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:15.105367 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" event={"ID":"b0899605-b3ca-4323-b44f-b60c475b64fc","Type":"ContainerStarted","Data":"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65"} Nov 03 10:06:15.105654 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:15.105471 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:15.222911 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:15.222860 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" podStartSLOduration=2.433995306 podStartE2EDuration="3.222845863s" totalImagesPullingTime="788.850557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:06:15.222288781 +0000 UTC m=+5198.890329511" watchObservedRunningTime="2026-11-03 10:06:15.222845863 +0000 UTC m=+5198.890886602" Nov 03 10:06:16.383888 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:16.383852 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:18.093537 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:18.093505 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:18.568737 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:18.568707 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:20.045573 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:20.045539 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:20.182097 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:20.182062 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:20.374212 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:20.374131 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:21.116681 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:21.116651 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:23.106385 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:23.106348 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:27.998386 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:27.998349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:32.995530 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:32.995496 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:35.560837 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:06:35.560796 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:06:37.909299 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:37.909266 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:40.265027 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:40.264989 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:42.211026 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.210988 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:42.211511 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.211224 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" podUID="b0899605-b3ca-4323-b44f-b60c475b64fc" containerName="maas-api" containerID="cri-o://8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65" gracePeriod=30 Nov 03 10:06:42.302468 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.302436 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:42.461503 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.461434 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:42.558279 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.558247 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\") pod \"b0899605-b3ca-4323-b44f-b60c475b64fc\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " Nov 03 10:06:42.558279 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.558293 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\") pod \"b0899605-b3ca-4323-b44f-b60c475b64fc\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " Nov 03 10:06:42.558568 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.558386 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") pod \"b0899605-b3ca-4323-b44f-b60c475b64fc\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " Nov 03 10:06:42.558568 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.558445 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\" (UniqueName: \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\") pod \"b0899605-b3ca-4323-b44f-b60c475b64fc\" (UID: \"b0899605-b3ca-4323-b44f-b60c475b64fc\") " Nov 03 10:06:42.558733 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.558699 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca" pod "b0899605-b3ca-4323-b44f-b60c475b64fc" (UID: "b0899605-b3ca-4323-b44f-b60c475b64fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:06:42.559392 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.559359 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0899605-b3ca-4323-b44f-b60c475b64fc-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:42.560797 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.560776 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls" pod "b0899605-b3ca-4323-b44f-b60c475b64fc" (UID: "b0899605-b3ca-4323-b44f-b60c475b64fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:06:42.560897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.560807 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs" pod "b0899605-b3ca-4323-b44f-b60c475b64fc" (UID: "b0899605-b3ca-4323-b44f-b60c475b64fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:06:42.560897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.560817 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x" pod "b0899605-b3ca-4323-b44f-b60c475b64fc" (UID: "b0899605-b3ca-4323-b44f-b60c475b64fc"). InnerVolumeSpecName "kube-api-access-ccn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:06:42.660337 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.660299 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:42.660337 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.660331 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccn2x\" (UniqueName: \"kubernetes.io/projected/b0899605-b3ca-4323-b44f-b60c475b64fc-kube-api-access-ccn2x\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:42.660337 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:42.660341 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0899605-b3ca-4323-b44f-b60c475b64fc-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:06:43.258413 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.258378 4543 generic.go:356] "Generic (PLEG): container finished" podID="b0899605-b3ca-4323-b44f-b60c475b64fc" containerID="8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65" exitCode=0 Nov 03 10:06:43.258860 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.258429 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" Nov 03 10:06:43.258860 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.258461 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" event={"ID":"b0899605-b3ca-4323-b44f-b60c475b64fc","Type":"ContainerDied","Data":"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65"} Nov 03 10:06:43.258860 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.258517 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7" event={"ID":"b0899605-b3ca-4323-b44f-b60c475b64fc","Type":"ContainerDied","Data":"dd05f98f43d2522bd1dc97470091a9e1ee7e2691c87404575c94889d8b3ebc27"} Nov 03 10:06:43.258860 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.258544 4543 scope.go:122] "RemoveContainer" containerID="8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65" Nov 03 10:06:43.268271 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.268256 4543 scope.go:122] "RemoveContainer" containerID="8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65" Nov 03 10:06:43.268528 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:06:43.268512 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65\": container with ID starting with 8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65 not found: ID does not exist" containerID="8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65" Nov 03 10:06:43.268583 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.268536 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65"} err="failed to get container status \"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65\": rpc error: code = NotFound desc = could not find container \"8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65\": container with ID starting with 8b29e7dac7fa37fcc67f5cff83d88b75acc2aee2bb0dec045884c6865f708f65 not found: ID does not exist" Nov 03 10:06:43.428771 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.428738 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:43.477585 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.477549 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fd27d2-df74445b4-d4kj7"] Nov 03 10:06:43.876104 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:43.876071 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:44.149352 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:44.149261 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:06:44.450532 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:44.450452 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0899605-b3ca-4323-b44f-b60c475b64fc" path="/var/lib/kubelet/pods/b0899605-b3ca-4323-b44f-b60c475b64fc/volumes" Nov 03 10:06:44.793399 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:06:44.793312 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:04.271853 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:04.271817 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:05.560659 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:07:05.560621 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:07:12.086166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:12.085228 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:07:12.086166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:12.085293 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:07:12.125293 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:12.125256 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:07:12.125634 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:12.125585 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:07:16.096534 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:16.096498 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:16.719636 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:16.719596 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:18.912694 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:18.912653 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:35.561705 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:07:35.561662 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:07:36.153089 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:36.153051 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:40.609408 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:40.609375 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:44.673343 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:44.673291 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:07:44.673881 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:44.673375 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:07:50.925951 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:50.925918 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:07:57.995469 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:07:57.995435 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:05.561371 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:08:05.561332 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:08:13.105166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:13.101479 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:08:13.105166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:13.101577 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:08:17.068164 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:17.068098 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:19.259596 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:19.259560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:21.315475 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:21.315435 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:35.561105 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:08:35.561062 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:08:38.202169 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:38.202119 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:40.953991 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:40.953954 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:08:57.086272 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:57.086224 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:08:57.086945 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:57.086910 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:08:57.129318 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:57.129285 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:08:57.133248 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:08:57.133211 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:09:05.560680 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:09:05.560632 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:09:17.418987 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:17.418947 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:09:19.600980 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:19.600940 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:09:35.560160 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:09:35.560097 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:09:37.207771 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:37.207741 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:09:37.262703 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:37.262670 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:09:41.306215 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:41.306165 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:09:46.354872 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:46.354833 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 10:09:51.355552 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.355477 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" podUID="8d4f8360-0c88-4514-83c5-797250d22890" containerName="payload-pre-processing" containerID="cri-o://d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c" gracePeriod=25 Nov 03 10:09:51.616750 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.616725 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 10:09:51.771804 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.771770 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\" (UniqueName: \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\") pod \"8d4f8360-0c88-4514-83c5-797250d22890\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " Nov 03 10:09:51.772003 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.771876 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\") pod \"8d4f8360-0c88-4514-83c5-797250d22890\" (UID: \"8d4f8360-0c88-4514-83c5-797250d22890\") " Nov 03 10:09:51.772270 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.772245 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume" pod "8d4f8360-0c88-4514-83c5-797250d22890" (UID: "8d4f8360-0c88-4514-83c5-797250d22890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:09:51.774103 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.774077 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg" pod "8d4f8360-0c88-4514-83c5-797250d22890" (UID: "8d4f8360-0c88-4514-83c5-797250d22890"). InnerVolumeSpecName "kube-api-access-qp9lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:09:51.873665 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.873588 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4f8360-0c88-4514-83c5-797250d22890-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:09:51.873665 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:51.873616 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9lg\" (UniqueName: \"kubernetes.io/projected/8d4f8360-0c88-4514-83c5-797250d22890-kube-api-access-qp9lg\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:09:52.435201 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.435163 4543 generic.go:356] "Generic (PLEG): container finished" podID="8d4f8360-0c88-4514-83c5-797250d22890" containerID="d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c" exitCode=0 Nov 03 10:09:52.435645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.435223 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" Nov 03 10:09:52.435645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.435229 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" event={"ID":"8d4f8360-0c88-4514-83c5-797250d22890","Type":"ContainerDied","Data":"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c"} Nov 03 10:09:52.435645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.435282 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f" event={"ID":"8d4f8360-0c88-4514-83c5-797250d22890","Type":"ContainerDied","Data":"d352c1578713d7fcae480d0011c2a1fe88dd0a03dd23527da0ddd55f5e08ecf7"} Nov 03 10:09:52.435645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.435312 4543 scope.go:122] "RemoveContainer" containerID="d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c" Nov 03 10:09:52.446559 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.446537 4543 scope.go:122] "RemoveContainer" containerID="d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c" Nov 03 10:09:52.447049 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:09:52.447028 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c\": container with ID starting with d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c not found: ID does not exist" containerID="d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c" Nov 03 10:09:52.447103 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.447059 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c"} err="failed to get container status \"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c\": rpc error: code = NotFound desc = could not find container \"d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c\": container with ID starting with d3d1edae2b51e7b9577b763732850efd258eb651ffe034ada6261eedecec5b4c not found: ID does not exist" Nov 03 10:09:52.604100 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.604058 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 10:09:52.654121 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:52.654088 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-r2s2f"] Nov 03 10:09:54.449533 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:09:54.449500 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4f8360-0c88-4514-83c5-797250d22890" path="/var/lib/kubelet/pods/8d4f8360-0c88-4514-83c5-797250d22890/volumes" Nov 03 10:10:00.438483 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:00.438443 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 10:10:05.439528 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.439473 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" podUID="6b6ed322-bd89-4c55-b650-28d37f9f208e" containerName="payload-processing" containerID="cri-o://4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d" gracePeriod=25 Nov 03 10:10:05.562603 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:10:05.562562 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:10:05.698379 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.698321 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 10:10:05.714528 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.714502 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\" (UniqueName: \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\") pod \"6b6ed322-bd89-4c55-b650-28d37f9f208e\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " Nov 03 10:10:05.714626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.714591 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\") pod \"6b6ed322-bd89-4c55-b650-28d37f9f208e\" (UID: \"6b6ed322-bd89-4c55-b650-28d37f9f208e\") " Nov 03 10:10:05.714964 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.714939 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume" pod "6b6ed322-bd89-4c55-b650-28d37f9f208e" (UID: "6b6ed322-bd89-4c55-b650-28d37f9f208e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:10:05.716703 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.716680 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s" pod "6b6ed322-bd89-4c55-b650-28d37f9f208e" (UID: "6b6ed322-bd89-4c55-b650-28d37f9f208e"). InnerVolumeSpecName "kube-api-access-kxn2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:10:05.815661 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.815630 4543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6ed322-bd89-4c55-b650-28d37f9f208e-plugins-config-volume\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:05.815661 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:05.815658 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxn2s\" (UniqueName: \"kubernetes.io/projected/6b6ed322-bd89-4c55-b650-28d37f9f208e-kube-api-access-kxn2s\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:10:06.502958 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.502927 4543 generic.go:356] "Generic (PLEG): container finished" podID="6b6ed322-bd89-4c55-b650-28d37f9f208e" containerID="4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d" exitCode=0 Nov 03 10:10:06.503414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.502978 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" Nov 03 10:10:06.503414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.502982 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" event={"ID":"6b6ed322-bd89-4c55-b650-28d37f9f208e","Type":"ContainerDied","Data":"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d"} Nov 03 10:10:06.503414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.503112 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lj9bz" event={"ID":"6b6ed322-bd89-4c55-b650-28d37f9f208e","Type":"ContainerDied","Data":"3f1c762662796e2b81953fd58784826ca3b03bf1e32c6cdb961da571d0cd61be"} Nov 03 10:10:06.503414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.503155 4543 scope.go:122] "RemoveContainer" containerID="4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d" Nov 03 10:10:06.511858 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.511839 4543 scope.go:122] "RemoveContainer" containerID="4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d" Nov 03 10:10:06.512131 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:10:06.512113 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d\": container with ID starting with 4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d not found: ID does not exist" containerID="4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d" Nov 03 10:10:06.512216 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.512159 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d"} err="failed to get container status \"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d\": rpc error: code = NotFound desc = could not find container \"4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d\": container with ID starting with 4610ebf4a813e0b7592f7d4d0f2ed5cb57bf27ca24576fcb4fc7a8544603cf0d not found: ID does not exist" Nov 03 10:10:06.671879 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.671843 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 10:10:06.676669 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:06.676644 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lj9bz"] Nov 03 10:10:08.450749 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:08.450716 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6ed322-bd89-4c55-b650-28d37f9f208e" path="/var/lib/kubelet/pods/6b6ed322-bd89-4c55-b650-28d37f9f208e/volumes" Nov 03 10:10:17.756593 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:17.756516 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:10:18.217886 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:18.217846 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:10:18.720244 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:18.720203 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:10:21.466289 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:21.466257 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:10:35.565393 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:10:35.565353 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:10:41.701052 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:10:41.701014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:11:05.561502 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:11:05.561457 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:11:18.097645 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:11:18.097609 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:11:35.564282 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:11:35.564239 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:11:42.003352 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:11:42.003314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:05.345278 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:05.345194 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:05.560307 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:12:05.560273 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:12:16.927867 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:16.927827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:17.059375 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:17.059338 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:17.452698 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:17.452660 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:18.439351 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:18.439315 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:12:35.561088 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:12:35.561052 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:12:42.341096 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:12:42.341064 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:13:05.559851 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:13:05.559815 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:13:18.778535 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:13:18.778446 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:13:35.561204 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:13:35.561167 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:13:42.708663 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:13:42.708611 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:13:44.666381 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:13:44.666331 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:13:44.667110 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:13:44.667064 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:14:05.560753 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:14:05.560705 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:14:19.128247 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:19.128216 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:14:28.085006 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:28.084961 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:14:35.561040 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:14:35.561003 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:14:37.237601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:37.237574 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:14:37.304700 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:37.304677 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:14:42.143165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:42.142238 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:14:42.143165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:42.142317 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:14:43.052763 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:14:43.052713 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:15:05.560841 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:15:05.560749 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:15:19.503517 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:15:19.503485 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:15:35.561592 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:15:35.561553 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:16:05.576439 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:16:05.576393 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:16:19.860806 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:16:19.860721 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:16:20.614431 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:16:20.614399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:16:35.782764 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:16:35.782719 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:16:55.226020 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:16:55.225982 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:17:05.708280 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:17:05.708239 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:17:20.987480 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:17:20.987442 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:17:30.145601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:17:30.145560 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:17:35.561208 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:17:35.561134 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:17:57.001789 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:17:57.001709 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:17:57.633794 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:17:57.633760 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:17:57.840446 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:17:57.840407 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:01.855364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:01.855329 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:02.029277 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:02.029242 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:02.168297 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:02.168263 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:05.235379 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:05.235347 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:05.393827 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:05.393791 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:05.560980 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:18:05.560892 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:18:08.152275 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152242 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:18:08.152718 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152704 4543 cpu_manager.go:414] "containerMap: removing container" podUID="8d4f8360-0c88-4514-83c5-797250d22890" containerName="payload-pre-processing" Nov 03 10:18:08.152765 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152719 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4f8360-0c88-4514-83c5-797250d22890" containerName="payload-pre-processing" Nov 03 10:18:08.152765 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152730 4543 cpu_manager.go:414] "containerMap: removing container" podUID="b0899605-b3ca-4323-b44f-b60c475b64fc" containerName="maas-api" Nov 03 10:18:08.152765 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152735 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0899605-b3ca-4323-b44f-b60c475b64fc" containerName="maas-api" Nov 03 10:18:08.152765 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152754 4543 cpu_manager.go:414] "containerMap: removing container" podUID="6b6ed322-bd89-4c55-b650-28d37f9f208e" containerName="payload-processing" Nov 03 10:18:08.152765 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152758 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6ed322-bd89-4c55-b650-28d37f9f208e" containerName="payload-processing" Nov 03 10:18:08.152928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152827 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6ed322-bd89-4c55-b650-28d37f9f208e" containerName="payload-processing" Nov 03 10:18:08.152928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152841 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4f8360-0c88-4514-83c5-797250d22890" containerName="payload-pre-processing" Nov 03 10:18:08.152928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.152848 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0899605-b3ca-4323-b44f-b60c475b64fc" containerName="maas-api" Nov 03 10:18:08.155902 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.155885 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.260719 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.260688 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-65cb27-serving-cert\"" Nov 03 10:18:08.262893 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.262865 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:18:08.290872 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.290834 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.290872 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.290879 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.291181 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.290985 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbbr\" (UniqueName: \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.291181 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.291020 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.345399 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.345366 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:08.391919 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.391887 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.392101 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.391929 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.392101 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.391964 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbbr\" (UniqueName: \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.392101 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.391988 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.392748 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.392684 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.394663 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.394640 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.394663 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.394653 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.451507 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.451438 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbbr\" (UniqueName: \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\") pod \"maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.469360 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.469333 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:08.506965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.506936 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:08.634321 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.634284 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:08.845658 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.845632 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:18:08.847694 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:18:08.847651 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c098dd_27c3_457c_92a4_f17df200fe11.slice/crio-465e4037758e1e5aa21768a1ae67b094971ab35f6ad977808b64906b7ea99219 WatchSource:0}: Error finding container 465e4037758e1e5aa21768a1ae67b094971ab35f6ad977808b64906b7ea99219: Status 404 returned error can't find the container with id 465e4037758e1e5aa21768a1ae67b094971ab35f6ad977808b64906b7ea99219 Nov 03 10:18:08.849204 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:08.849187 4543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 10:18:09.202625 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:09.202579 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" event={"ID":"06c098dd-27c3-457c-92a4-f17df200fe11","Type":"ContainerStarted","Data":"465e4037758e1e5aa21768a1ae67b094971ab35f6ad977808b64906b7ea99219"} Nov 03 10:18:10.209151 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:10.209106 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" event={"ID":"06c098dd-27c3-457c-92a4-f17df200fe11","Type":"ContainerStarted","Data":"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a"} Nov 03 10:18:10.209553 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:10.209224 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:10.328247 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:10.328177 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" podStartSLOduration=1.390620249 podStartE2EDuration="2.328156421s" totalImagesPullingTime="937.536172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:18:10.325788622 +0000 UTC m=+5913.993829426" watchObservedRunningTime="2026-11-03 10:18:10.328156421 +0000 UTC m=+5913.996197155" Nov 03 10:18:13.496840 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:13.496806 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:16.219867 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:16.219840 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:18:18.501454 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:18.501411 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:21.320840 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:21.320765 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:23.490716 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:23.490682 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:26.752443 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:26.752406 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:27.395358 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:27.395323 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:27.534247 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:27.534214 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:27.665719 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:27.665629 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:28.890317 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:28.890272 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:29.958377 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:29.958339 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:30.495453 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:30.495420 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:30.638688 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:30.638626 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:33.897795 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:33.897754 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:35.561162 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:18:35.561109 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:18:38.904463 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:38.904414 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:43.820285 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:43.820247 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:48.807867 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:48.807827 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:53.710980 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:53.710948 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:57.848723 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:57.848689 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:57.977506 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:57.977471 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:18:58.701421 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:18:58.701383 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:03.688343 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:03.688304 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:05.561253 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:19:05.561125 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:19:08.705705 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:08.705669 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:13.697853 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:13.697808 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:18.705989 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:18.705904 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:23.703523 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:23.703490 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:27.304333 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:27.304298 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:28.712312 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:28.712276 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:28.847081 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:28.847046 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:30.837440 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:30.837407 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:31.664318 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:31.664287 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:32.401730 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:32.401690 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:33.706868 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:33.706831 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:35.523084 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:35.523052 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:35.561106 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:19:35.561071 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:19:36.811206 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:36.811127 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:36.945383 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:36.945349 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:37.269496 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:37.269471 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:19:37.339567 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:37.339542 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:19:38.197430 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:38.197396 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:19:38.197954 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:38.197648 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" podUID="06c098dd-27c3-457c-92a4-f17df200fe11" containerName="maas-api" containerID="cri-o://aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a" gracePeriod=30 Nov 03 10:19:38.297062 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:38.297027 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:39.451375 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.451347 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:19:39.597671 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.597588 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\") pod \"06c098dd-27c3-457c-92a4-f17df200fe11\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " Nov 03 10:19:39.597671 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.597656 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\") pod \"06c098dd-27c3-457c-92a4-f17df200fe11\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " Nov 03 10:19:39.597951 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.597692 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\") pod \"06c098dd-27c3-457c-92a4-f17df200fe11\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " Nov 03 10:19:39.597951 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.597834 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\" (UniqueName: \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\") pod \"06c098dd-27c3-457c-92a4-f17df200fe11\" (UID: \"06c098dd-27c3-457c-92a4-f17df200fe11\") " Nov 03 10:19:39.598102 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.598067 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca" pod "06c098dd-27c3-457c-92a4-f17df200fe11" (UID: "06c098dd-27c3-457c-92a4-f17df200fe11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:19:39.598246 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.598231 4543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c098dd-27c3-457c-92a4-f17df200fe11-openshift-service-ca\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:39.600506 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.600482 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs" pod "06c098dd-27c3-457c-92a4-f17df200fe11" (UID: "06c098dd-27c3-457c-92a4-f17df200fe11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:19:39.600616 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.600504 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls" pod "06c098dd-27c3-457c-92a4-f17df200fe11" (UID: "06c098dd-27c3-457c-92a4-f17df200fe11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 10:19:39.600689 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.600620 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr" pod "06c098dd-27c3-457c-92a4-f17df200fe11" (UID: "06c098dd-27c3-457c-92a4-f17df200fe11"). InnerVolumeSpecName "kube-api-access-nlbbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:19:39.647571 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.647541 4543 generic.go:356] "Generic (PLEG): container finished" podID="06c098dd-27c3-457c-92a4-f17df200fe11" containerID="aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a" exitCode=0 Nov 03 10:19:39.647727 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.647593 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" Nov 03 10:19:39.647727 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.647601 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" event={"ID":"06c098dd-27c3-457c-92a4-f17df200fe11","Type":"ContainerDied","Data":"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a"} Nov 03 10:19:39.647727 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.647643 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2" event={"ID":"06c098dd-27c3-457c-92a4-f17df200fe11","Type":"ContainerDied","Data":"465e4037758e1e5aa21768a1ae67b094971ab35f6ad977808b64906b7ea99219"} Nov 03 10:19:39.647727 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.647664 4543 scope.go:122] "RemoveContainer" containerID="aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a" Nov 03 10:19:39.657783 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.657760 4543 scope.go:122] "RemoveContainer" containerID="aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a" Nov 03 10:19:39.658090 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:19:39.658070 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a\": container with ID starting with aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a not found: ID does not exist" containerID="aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a" Nov 03 10:19:39.658183 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.658100 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a"} err="failed to get container status \"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a\": rpc error: code = NotFound desc = could not find container \"aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a\": container with ID starting with aecd07e81a44552405ff6c10184622a7fc00022f38dd1910e51c786967fd8e5a not found: ID does not exist" Nov 03 10:19:39.699650 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.699620 4543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-maas-api-tls\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:39.699650 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.699645 4543 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c098dd-27c3-457c-92a4-f17df200fe11-metrics-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:39.699819 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.699655 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbbr\" (UniqueName: \"kubernetes.io/projected/06c098dd-27c3-457c-92a4-f17df200fe11-kube-api-access-nlbbr\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:19:39.815654 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.815622 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:19:39.820927 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:39.820903 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-65cb27-fd97b5cd4-nkrf2"] Nov 03 10:19:40.450701 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:40.450656 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c098dd-27c3-457c-92a4-f17df200fe11" path="/var/lib/kubelet/pods/06c098dd-27c3-457c-92a4-f17df200fe11/volumes" Nov 03 10:19:41.721891 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:41.721855 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:48.179859 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:48.179824 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:48.311943 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:48.311903 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:53.618490 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:53.618453 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:54.360568 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:54.360532 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:56.286349 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:56.286314 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:19:56.922908 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:19:56.922871 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:20:00.721481 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:20:00.721441 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:20:00.854417 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:20:00.854370 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:20:05.561808 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:20:05.561763 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:20:31.189023 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:20:31.188985 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:20:32.004057 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:20:32.004025 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:20:35.560813 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:20:35.560778 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:20:48.342161 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:20:48.342047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:21:05.561029 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:21:05.560986 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:21:31.588880 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:21:31.588843 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:21:32.345088 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:21:32.345047 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:21:35.560090 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:21:35.560051 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:22:05.576894 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:22:05.576812 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:22:05.859080 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:05.858991 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:22:31.923919 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:31.923835 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:22:32.686910 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:32.686876 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:22:35.560883 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:22:35.560832 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:22:53.118191 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:53.118132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:22:53.362353 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:53.362317 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:22:53.998053 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:22:53.998014 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:23:05.560794 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:23:05.560753 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:23:32.298937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:23:32.298902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:23:33.034869 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:23:33.034834 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:23:35.560814 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:23:35.560775 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:23:58.094246 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:23:58.092378 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:23:58.094246 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:23:58.092451 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:24:05.560791 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:24:05.560751 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:24:32.560347 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:32.560298 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:24:33.482005 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:33.481972 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:24:35.561434 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:24:35.561398 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:24:37.299876 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:37.299844 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:24:37.372099 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:37.372071 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:24:59.662366 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:59.662319 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:24:59.663171 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:24:59.663110 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:25:05.565960 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:25:05.565919 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:25:06.994364 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:06.994300 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:25:07.637623 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:07.637548 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:25:12.077166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:12.073310 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:25:12.077166 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:12.073403 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:25:12.125165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:12.124238 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:25:12.125165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:12.124304 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:25:32.903478 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:32.903373 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:25:33.724354 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:25:33.724315 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:25:35.670957 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:25:35.670868 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:26:05.560444 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:26:05.560402 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:26:28.821489 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:26:28.821455 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:26:33.294870 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:26:33.294790 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:26:35.560042 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:26:35.560001 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:27:03.153484 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:03.153399 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:27:05.560202 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:27:05.560160 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:27:33.586479 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:33.586408 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:27:34.409254 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:34.409219 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:27:35.561159 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:27:35.561101 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:27:43.100165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:43.099951 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 10:27:43.100165 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:43.100027 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 10:27:44.779444 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:27:44.779383 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:28:05.561115 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:28:05.561074 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:28:17.014764 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:28:17.014648 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:28:33.933608 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:28:33.933575 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:28:34.768161 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:28:34.767210 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:28:35.721675 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:28:35.721590 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:28:45.127846 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:28:45.127813 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:29:05.560809 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:29:05.560772 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:29:30.062575 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:30.062544 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:29:30.413733 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:30.413705 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:29:31.666519 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:31.666487 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:29:32.246705 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:32.246674 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:29:32.470039 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:32.470008 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:29:33.765917 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:33.765884 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-75bf4769fd-bcvvb_df03d5cf-0aa6-4f48-aa72-eac45b12c185/maas-api/0.log" Nov 03 10:29:34.278192 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:34.278132 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:29:35.560442 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:29:35.560405 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:29:37.330910 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:37.330884 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:29:37.412242 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:37.412213 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:29:45.449683 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:45.449645 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:29:49.815437 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:49.815360 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:29:50.101441 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:50.101365 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:29:51.169167 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:51.169122 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:29:52.893473 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:52.893439 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:29:53.331031 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:53.330953 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:29:53.652420 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:53.652392 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:29:54.810068 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:54.810036 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:29:55.418293 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:55.418254 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:29:57.980264 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:57.980231 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:29:58.246603 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:58.246530 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:29:58.713693 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:58.713660 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:29:59.208343 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:29:59.208314 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:30:05.561546 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:30:05.561509 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:30:09.790264 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:09.790227 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:09.921109 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:09.921076 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:19.243267 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:19.243235 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qkzjz_f61e725e-6128-4977-9ac4-ee0df1feb8db/global-pull-secret-syncer/0.log" Nov 03 10:30:21.766121 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:21.766085 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" Nov 03 10:30:22.426289 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:22.426254 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:30:22.666112 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:22.666086 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:30:23.531749 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:23.531720 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-117.us-west-1.compute.internal_20fdfbfab5aa17ccf7324ae585a08734/haproxy/0.log" Nov 03 10:30:24.286914 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:24.286878 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:30:24.531224 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:24.531194 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:30:25.243632 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:25.243597 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:30:28.378550 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:28.378519 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:30:28.717100 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:28.717023 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:30:30.205476 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:30.205446 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:30:30.565127 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:30.565053 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:30:31.282720 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:31.282686 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:30:34.463091 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:34.463052 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:30:34.610778 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:34.610749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:34.802118 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:34.802046 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:30:35.450515 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:35.450481 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:35.561321 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:30:35.561282 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:30:36.307168 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:36.307108 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:30:36.669103 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:36.669075 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:30:37.384846 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:37.384812 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:30:42.541836 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:42.541805 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:30:42.978821 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:42.978794 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:30:43.198036 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:43.198010 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:30:44.071335 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:44.071305 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:30:44.509355 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:44.509330 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:30:45.491597 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:45.491563 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:45.803981 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:45.803902 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:30:48.473181 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:48.473130 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:30:48.899327 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:48.899297 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:30:50.235676 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:50.235638 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:30:51.254715 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:51.254682 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:30:51.470066 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:51.470040 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:30:52.182229 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:52.182195 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:30:56.603381 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:56.603345 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wqht9_9edb5966-dc9a-4536-bfa9-bcfe3df2baf1/kube-state-metrics/0.log" Nov 03 10:30:56.997258 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:56.997224 4543 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c"] Nov 03 10:30:56.997821 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:56.997802 4543 cpu_manager.go:414] "containerMap: removing container" podUID="06c098dd-27c3-457c-92a4-f17df200fe11" containerName="maas-api" Nov 03 10:30:56.997893 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:56.997823 4543 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c098dd-27c3-457c-92a4-f17df200fe11" containerName="maas-api" Nov 03 10:30:56.997936 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:56.997899 4543 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c098dd-27c3-457c-92a4-f17df200fe11" containerName="maas-api" Nov 03 10:30:57.001052 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.001037 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.022947 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.022921 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wqht9_9edb5966-dc9a-4536-bfa9-bcfe3df2baf1/kube-rbac-proxy-main/0.log" Nov 03 10:30:57.057679 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.057651 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c"] Nov 03 10:30:57.105637 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.105611 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"kube-root-ca.crt\"" Nov 03 10:30:57.105800 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.105690 4543 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b4k9m\"/\"default-dockercfg-xt4bq\"" Nov 03 10:30:57.105800 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.105735 4543 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"openshift-service-ca.crt\"" Nov 03 10:30:57.122715 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.122693 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-sys\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.122876 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.122768 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rhs\" (UniqueName: \"kubernetes.io/projected/41b55aea-1f52-4be8-ae81-4e5cbea59d80-kube-api-access-s4rhs\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.122876 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.122855 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-podres\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.122995 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.122949 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-lib-modules\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.122995 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.122989 4543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-proc\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224578 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224544 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rhs\" (UniqueName: \"kubernetes.io/projected/41b55aea-1f52-4be8-ae81-4e5cbea59d80-kube-api-access-s4rhs\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224578 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224583 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-podres\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224648 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-lib-modules\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224683 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-proc\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224721 4543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-sys\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224762 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-podres\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224774 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-lib-modules\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.224842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224776 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-proc\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.225059 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.224862 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41b55aea-1f52-4be8-ae81-4e5cbea59d80-sys\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.283894 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.283825 4543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rhs\" (UniqueName: \"kubernetes.io/projected/41b55aea-1f52-4be8-ae81-4e5cbea59d80-kube-api-access-s4rhs\") pod \"perf-node-gather-daemonset-jhb6c\" (UID: \"41b55aea-1f52-4be8-ae81-4e5cbea59d80\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.313909 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.313880 4543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:57.445367 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.445337 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-wqht9_9edb5966-dc9a-4536-bfa9-bcfe3df2baf1/kube-rbac-proxy-self/0.log" Nov 03 10:30:57.683999 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.683975 4543 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c"] Nov 03 10:30:57.686200 ip-10-0-131-117 kubenswrapper[4543]: W1103 10:30:57.686168 4543 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41b55aea_1f52_4be8_ae81_4e5cbea59d80.slice/crio-675006cd0ef2fccf2d88f5fef30874e6bc6506f088146d2b670e1f8181a5b25a WatchSource:0}: Error finding container 675006cd0ef2fccf2d88f5fef30874e6bc6506f088146d2b670e1f8181a5b25a: Status 404 returned error can't find the container with id 675006cd0ef2fccf2d88f5fef30874e6bc6506f088146d2b670e1f8181a5b25a Nov 03 10:30:57.891403 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.891362 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" event={"ID":"41b55aea-1f52-4be8-ae81-4e5cbea59d80","Type":"ContainerStarted","Data":"0a61a63c2b9dc8e2fb02ca413dbcaa7315a60f35d650a3f689bc23b29740e357"} Nov 03 10:30:57.891595 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.891421 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" event={"ID":"41b55aea-1f52-4be8-ae81-4e5cbea59d80","Type":"ContainerStarted","Data":"675006cd0ef2fccf2d88f5fef30874e6bc6506f088146d2b670e1f8181a5b25a"} Nov 03 10:30:57.891595 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:57.891516 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:30:58.009555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:30:58.009450 4543 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" podStartSLOduration=2.009431961 podStartE2EDuration="2.009431961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:58.007127668 +0000 UTC m=+6681.675168434" watchObservedRunningTime="2026-11-03 10:30:58.009431961 +0000 UTC m=+6681.677472701" Nov 03 10:31:01.285363 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:01.285318 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttb6z_13dadf9a-ed46-4f0e-8bbc-5925697bb5f8/node-exporter/0.log" Nov 03 10:31:01.709330 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:01.709297 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttb6z_13dadf9a-ed46-4f0e-8bbc-5925697bb5f8/kube-rbac-proxy/0.log" Nov 03 10:31:02.132225 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:02.132195 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ttb6z_13dadf9a-ed46-4f0e-8bbc-5925697bb5f8/init-textfile/0.log" Nov 03 10:31:02.559189 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:02.559060 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-f6spt_6eff5b5c-c34c-422b-9a8e-ce973b09c18c/kube-rbac-proxy-main/0.log" Nov 03 10:31:02.981200 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:02.981123 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-f6spt_6eff5b5c-c34c-422b-9a8e-ce973b09c18c/kube-rbac-proxy-self/0.log" Nov 03 10:31:03.404435 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:03.404396 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-f6spt_6eff5b5c-c34c-422b-9a8e-ce973b09c18c/openshift-state-metrics/0.log" Nov 03 10:31:03.837642 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:03.837561 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/prometheus/0.log" Nov 03 10:31:03.907314 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:03.907290 4543 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-jhb6c" Nov 03 10:31:04.259351 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:04.259325 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/config-reloader/0.log" Nov 03 10:31:04.682906 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:04.682875 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/thanos-sidecar/0.log" Nov 03 10:31:05.104998 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:05.104925 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/kube-rbac-proxy-web/0.log" Nov 03 10:31:05.526481 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:05.526451 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/kube-rbac-proxy/0.log" Nov 03 10:31:05.560838 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:31:05.560796 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:31:05.949004 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:05.948971 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/kube-rbac-proxy-thanos/0.log" Nov 03 10:31:06.373238 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:06.373203 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f8043a05-3979-4dd3-9a36-5bf3d14ed924/init-config-reloader/0.log" Nov 03 10:31:17.886578 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:17.886456 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-hhhdr_47ce01da-7ee4-4fe5-af89-f5953af91c65/cert-manager-controller/0.log" Nov 03 10:31:18.103561 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:18.103515 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-nm7k4_d53304b8-5e14-4e3a-82b9-c4d4799ddd09/cert-manager-cainjector/0.log" Nov 03 10:31:20.049463 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:20.049436 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-7gbhp_e6de53d9-dec4-4b9a-a826-0d3a3e8d37f8/download-server/0.log" Nov 03 10:31:20.578387 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:20.578360 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-4sfqv_e676bda6-83ca-4fd4-920e-67fe53e41f16/manager/0.log" Nov 03 10:31:20.791238 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:20.791204 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-4sfqv_e676bda6-83ca-4fd4-920e-67fe53e41f16/copy-manifests/0.log" Nov 03 10:31:23.992455 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:23.992422 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:31:24.331621 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:24.331533 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:31:25.869679 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:25.869641 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:31:26.181172 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:26.181075 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:31:26.901886 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:26.901854 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:31:28.311374 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:28.311342 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45lmv_7b492e70-992c-4975-9153-dbc89048db4b/dns/0.log" Nov 03 10:31:28.733795 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:28.733761 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45lmv_7b492e70-992c-4975-9153-dbc89048db4b/kube-rbac-proxy/0.log" Nov 03 10:31:30.727976 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:30.727950 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:31:31.165583 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:31.165549 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:31:31.283877 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:31.283852 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pr47_2f00260d-22ef-44e7-802c-d1b9b9585b61/dns-node-resolver/0.log" Nov 03 10:31:31.383723 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:31.383697 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:31:32.257524 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:32.257493 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:31:32.690951 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:32.690920 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:31:34.571555 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:34.571522 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69656f6bbf-4j7qp_d06ec521-31d5-4f36-ae0e-27efc1c0a1dd/registry/0.log" Nov 03 10:31:34.959160 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:34.959110 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:31:35.540650 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:35.540618 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dfmxv_a57c4f2b-51b2-43d1-9fa4-052405923764/node-ca/0.log" Nov 03 10:31:35.561590 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:31:35.561556 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:31:35.817790 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:35.817707 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:31:37.678800 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:37.678768 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-vp488_d2d32aad-0569-4d35-a08c-44d13f35fe28/jobset-operator/0.log" Nov 03 10:31:41.944128 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:41.944103 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:31:42.406180 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:42.406155 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:31:42.639950 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:42.639921 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:31:43.536350 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:43.536323 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:31:43.587120 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:43.587094 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:31:43.969477 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:43.969444 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:31:44.439928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:44.439900 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:31:44.866120 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:44.866095 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:31:46.168275 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:46.168211 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:31:46.582198 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:46.582115 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:31:47.444150 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:47.444101 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:31:51.602571 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:51.602542 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-6w6tg_2df8810e-ad76-452e-9da1-581d58a9b9b7/kube-auth-proxy/0.log" Nov 03 10:31:52.045355 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:52.045327 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-7s6wh_d1907c83-2863-4e88-8995-356912ee98a2/istio-proxy/0.log" Nov 03 10:31:52.265546 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:52.265510 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9tmmz_bec7421a-e681-494c-989c-e780b975158f/istio-proxy/0.log" Nov 03 10:31:53.147592 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:53.147560 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gtk7l_13546769-d45e-45c9-be20-621f4f329f79/istio-proxy/0.log" Nov 03 10:31:53.585987 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:53.585881 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-h9rgl_a37aaef4-8a1f-4f2d-a97e-558328072820/istio-proxy/0.log" Nov 03 10:31:54.007267 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:54.007211 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-678m8_0ae96a70-69ca-4de6-a9f5-8b7202bffba8/serve-healthcheck-canary/0.log" Nov 03 10:31:59.029022 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:59.028994 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd45j_1c816c18-0b4b-42a4-95cd-dde80e6e75bd/kube-rbac-proxy/0.log" Nov 03 10:31:59.452097 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:59.452064 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd45j_1c816c18-0b4b-42a4-95cd-dde80e6e75bd/exporter/0.log" Nov 03 10:31:59.876002 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:31:59.875957 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pd45j_1c816c18-0b4b-42a4-95cd-dde80e6e75bd/extractor/0.log" Nov 03 10:32:05.560470 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:32:05.560434 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:32:06.353837 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:06.353784 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-4sfqv_e676bda6-83ca-4fd4-920e-67fe53e41f16/manager/0.log" Nov 03 10:32:06.777020 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:06.776987 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-4sfqv_e676bda6-83ca-4fd4-920e-67fe53e41f16/copy-manifests/0.log" Nov 03 10:32:12.078927 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:12.078895 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-vp488_d2d32aad-0569-4d35-a08c-44d13f35fe28/jobset-operator/0.log" Nov 03 10:32:14.624792 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:14.624759 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58b44ff57b-z4gsf_e46eb3d7-9990-4e70-b44b-be4a24105698/manager/0.log" Nov 03 10:32:15.175367 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:15.175337 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n64sg_0124bfb9-f59d-47cf-904b-041308f81dca/manager/0.log" Nov 03 10:32:15.290187 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:15.290159 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:32:15.505767 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:15.505697 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:32:16.217101 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:16.217067 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:32:17.094477 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:17.094446 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-kfmgw_3f4062a6-2fa5-49e0-ae6a-9f30f42d3e7f/manager/0.log" Nov 03 10:32:17.472154 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:17.472114 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-4s8bk_bec5a14a-dcb9-4022-8849-2b4553225759/server/0.log" Nov 03 10:32:18.415908 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:18.415870 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wg2dn_1cac33c5-bd20-499b-b8ec-78aee6e9faf8/manager/0.log" Nov 03 10:32:35.302367 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:35.302327 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:32:35.561192 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:32:35.561093 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:32:46.522982 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:46.522933 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:32:46.911599 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:46.911567 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4xgnk_518c5a4f-9809-452d-8a8d-aea90d5961c5/kube-multus/0.log" Nov 03 10:32:51.099523 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:51.099494 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/kube-multus-additional-cni-plugins/0.log" Nov 03 10:32:51.185377 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:51.185308 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:32:51.402776 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:51.402750 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:32:51.520710 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:51.520654 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/egress-router-binary-copy/0.log" Nov 03 10:32:51.942950 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:51.942903 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/cni-plugins/0.log" Nov 03 10:32:52.111290 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:52.111256 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:32:52.516008 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:52.515964 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/bond-cni-plugin/0.log" Nov 03 10:32:52.937543 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:52.937514 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/routeoverride-cni/0.log" Nov 03 10:32:53.361992 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:53.361968 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/whereabouts-cni-bincopy/0.log" Nov 03 10:32:53.784673 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:53.784587 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bj87v_15fbcb5f-5b60-488c-929a-e5661647fb3a/whereabouts-cni/0.log" Nov 03 10:32:57.807700 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:57.807666 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8jcr_bec20a64-e1e9-49fc-bb02-d62c451121e1/network-metrics-daemon/0.log" Nov 03 10:32:58.225842 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:32:58.225803 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d8jcr_bec20a64-e1e9-49fc-bb02-d62c451121e1/kube-rbac-proxy/0.log" Nov 03 10:33:03.470905 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:03.470878 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:33:03.687054 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:03.687024 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:33:04.119244 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:04.119215 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:33:04.553335 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:04.553248 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:33:05.560101 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:33:05.560065 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:33:11.661748 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:11.661719 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-controller/0.log" Nov 03 10:33:12.119928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:12.119905 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/1.log" Nov 03 10:33:12.119928 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:12.119926 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:33:12.439220 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:12.439106 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/kube-rbac-proxy-node/0.log" Nov 03 10:33:12.864524 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:12.864493 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/kube-rbac-proxy-ovn-metrics/0.log" Nov 03 10:33:13.287834 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:13.287755 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/northd/0.log" Nov 03 10:33:13.711522 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:13.711497 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/nbdb/0.log" Nov 03 10:33:14.133406 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:14.133384 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/sbdb/0.log" Nov 03 10:33:14.650304 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:14.650274 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovnkube-controller/0.log" Nov 03 10:33:22.242478 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:22.242445 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w4gf5_7d926adb-98d1-46bb-a1e9-8624624c0f2f/network-check-target-container/0.log" Nov 03 10:33:27.359793 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:27.359758 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:33:27.575177 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:27.575128 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:33:28.288468 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:28.288437 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:33:28.884915 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:28.884871 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-62fp4_530f4286-b8c2-4331-9bcc-372807e290d9/iptables-alerter/0.log" Nov 03 10:33:34.507158 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:34.507108 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-f4ldt_b8977ebc-ddf6-41df-9faf-175c92054114/tuned/0.log" Nov 03 10:33:35.560470 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:33:35.560429 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:33:37.164716 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:37.164680 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:33:37.289276 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:37.289241 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:33:41.371921 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:41.371875 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:33:41.580255 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:41.580227 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:33:42.017116 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:42.017076 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:33:42.450187 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:42.450126 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:33:46.867004 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:46.866968 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:33:53.862487 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:53.862442 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-xsqnh_42bbce16-077e-4e80-8085-26d639826e05/service-ca-controller/0.log" Nov 03 10:33:57.844965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:57.844915 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-k285j_8c823c7a-c6fb-4796-9a85-5f23e6c05759/csi-driver/0.log" Nov 03 10:33:58.267363 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:58.267314 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-k285j_8c823c7a-c6fb-4796-9a85-5f23e6c05759/csi-node-driver-registrar/0.log" Nov 03 10:33:58.690132 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:33:58.690088 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-k285j_8c823c7a-c6fb-4796-9a85-5f23e6c05759/csi-liveness-probe/0.log" Nov 03 10:34:03.731826 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:03.731796 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:34:03.947295 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:03.947265 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:34:04.661466 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:04.661430 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:34:05.560590 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:05.560553 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:34:06.856386 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:06.856358 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 10:34:06.856859 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:06.856553 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" containerID="cri-o://aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d" gracePeriod=30 Nov 03 10:34:11.081536 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:11.081495 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:34:11.081946 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:11.081553 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" podUID="13546769-d45e-45c9-be20-621f4f329f79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:34:12.121357 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.121330 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 10:34:12.234799 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234722 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.234799 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234766 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235029 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234810 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235029 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234839 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235029 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234872 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235029 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234940 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235029 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.234989 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235321 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235198 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:12.235321 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235216 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235321 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235222 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:12.235321 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235305 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\") pod \"13546769-d45e-45c9-be20-621f4f329f79\" (UID: \"13546769-d45e-45c9-be20-621f4f329f79\") " Nov 03 10:34:12.235550 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235428 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:12.235669 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235619 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:12.235827 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235685 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.235827 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235706 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.235827 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235666 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:12.235996 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235722 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.235996 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.235865 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.237410 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.237385 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:12.237528 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.237421 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "kube-api-access-j5s4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:12.237670 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.237642 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:12.237670 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.237651 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo" pod "13546769-d45e-45c9-be20-621f4f329f79" (UID: "13546769-d45e-45c9-be20-621f4f329f79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:12.336690 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.336654 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13546769-d45e-45c9-be20-621f4f329f79-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.336690 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.336683 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.336690 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.336695 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5s4z\" (UniqueName: \"kubernetes.io/projected/13546769-d45e-45c9-be20-621f4f329f79-kube-api-access-j5s4z\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.336941 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.336707 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13546769-d45e-45c9-be20-621f4f329f79-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.336941 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.336719 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13546769-d45e-45c9-be20-621f4f329f79-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:12.855105 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.855067 4543 generic.go:356] "Generic (PLEG): container finished" podID="13546769-d45e-45c9-be20-621f4f329f79" containerID="aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d" exitCode=0 Nov 03 10:34:12.855322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.855122 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" Nov 03 10:34:12.855322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.855171 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" event={"ID":"13546769-d45e-45c9-be20-621f4f329f79","Type":"ContainerDied","Data":"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d"} Nov 03 10:34:12.855322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.855231 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l" event={"ID":"13546769-d45e-45c9-be20-621f4f329f79","Type":"ContainerDied","Data":"013e6556959972bdab4cea76cf35e59b1ba5d518bc2b7f3691c596930a36ba2a"} Nov 03 10:34:12.855322 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.855255 4543 scope.go:122] "RemoveContainer" containerID="aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d" Nov 03 10:34:12.864071 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.864052 4543 scope.go:122] "RemoveContainer" containerID="aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d" Nov 03 10:34:12.864325 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:12.864305 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d\": container with ID starting with aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d not found: ID does not exist" containerID="aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d" Nov 03 10:34:12.864387 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:12.864335 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d"} err="failed to get container status \"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d\": rpc error: code = NotFound desc = could not find container \"aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d\": container with ID starting with aedd019dfc5c69bed9a97af5c0c130e1f70425fa4fe122ae2551f67d78027c2d not found: ID does not exist" Nov 03 10:34:13.077736 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:13.077700 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 10:34:13.127202 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:13.127170 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gtk7l"] Nov 03 10:34:14.451414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:14.451377 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13546769-d45e-45c9-be20-621f4f329f79" path="/var/lib/kubelet/pods/13546769-d45e-45c9-be20-621f4f329f79/volumes" Nov 03 10:34:21.989952 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:21.989917 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 10:34:21.990406 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:21.990201 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" containerID="cri-o://8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c" gracePeriod=30 Nov 03 10:34:22.039250 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:22.039211 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:34:22.039475 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:22.039438 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" podUID="bec7421a-e681-494c-989c-e780b975158f" containerName="istio-proxy" containerID="cri-o://8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243" gracePeriod=30 Nov 03 10:34:22.041104 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:22.041077 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:34:22.041331 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:22.041309 4543 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" containerID="cri-o://5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d" gracePeriod=30 Nov 03 10:34:26.032298 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:26.032261 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 10:34:26.032738 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:26.032313 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" podUID="d1907c83-2863-4e88-8995-356912ee98a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 10:34:27.083287 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.083231 4543 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": read tcp 10.134.0.2:49482->10.134.0.151:15021: read: connection reset by peer" start-of-body= Nov 03 10:34:27.083718 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.083303 4543 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": read tcp 10.134.0.2:49482->10.134.0.151:15021: read: connection reset by peer" Nov 03 10:34:27.306300 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.306271 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 10:34:27.315990 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.315966 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:34:27.347361 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.347328 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:34:27.481786 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481742 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.481965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481810 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.481965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481840 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.481965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481871 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.481965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481914 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.481965 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481949 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.481983 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482009 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482041 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482080 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482128 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482189 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.482303 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482248 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482306 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482333 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482370 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482404 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482431 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482471 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482474 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482498 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482539 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482540 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482567 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482604 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.482680 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482641 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\") pod \"bec7421a-e681-494c-989c-e780b975158f\" (UID: \"bec7421a-e681-494c-989c-e780b975158f\") " Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482692 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\") pod \"d1907c83-2863-4e88-8995-356912ee98a2\" (UID: \"d1907c83-2863-4e88-8995-356912ee98a2\") " Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482722 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482764 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482765 4543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\") pod \"a37aaef4-8a1f-4f2d-a97e-558328072820\" (UID: \"a37aaef4-8a1f-4f2d-a97e-558328072820\") " Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.482985 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.483027 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:27.483397 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.483209 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484115 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.483453 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:27.484115 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.483543 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484115 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.483967 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484115 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484089 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484457 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484200 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484545 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484514 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 10:34:27.484545 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484539 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484698 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484218 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.484862 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484841 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484869 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37aaef4-8a1f-4f2d-a97e-558328072820-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484883 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484897 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484910 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484924 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.484937 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484936 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1907c83-2863-4e88-8995-356912ee98a2-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484950 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484963 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484975 4543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec7421a-e681-494c-989c-e780b975158f-istiod-ca-cert\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.484988 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.485000 4543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-workload-certs\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.485012 4543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-data\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.485025 4543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-workload-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.485471 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.485272 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.485945 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.485720 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.487067 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487040 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.488015 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487792 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.488015 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487801 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "kube-api-access-s5bp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.488015 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487850 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.488015 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487869 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 10:34:27.488015 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.487983 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.488414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.488261 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "kube-api-access-b88b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.488564 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.488544 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo" pod "bec7421a-e681-494c-989c-e780b975158f" (UID: "bec7421a-e681-494c-989c-e780b975158f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:27.488681 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.488662 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:27.488750 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.488669 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo" pod "d1907c83-2863-4e88-8995-356912ee98a2" (UID: "d1907c83-2863-4e88-8995-356912ee98a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 10:34:27.489405 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.489385 4543 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md" pod "a37aaef4-8a1f-4f2d-a97e-558328072820" (UID: "a37aaef4-8a1f-4f2d-a97e-558328072820"). InnerVolumeSpecName "kube-api-access-tj9md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 10:34:27.586824 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586743 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.586824 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586785 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.586824 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586800 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.586824 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586814 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586847 4543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-token\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586860 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37aaef4-8a1f-4f2d-a97e-558328072820-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586874 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9md\" (UniqueName: \"kubernetes.io/projected/a37aaef4-8a1f-4f2d-a97e-558328072820-kube-api-access-tj9md\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586890 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bp5\" (UniqueName: \"kubernetes.io/projected/d1907c83-2863-4e88-8995-356912ee98a2-kube-api-access-s5bp5\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586904 4543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88b4\" (UniqueName: \"kubernetes.io/projected/bec7421a-e681-494c-989c-e780b975158f-kube-api-access-b88b4\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586918 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec7421a-e681-494c-989c-e780b975158f-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586933 4543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1907c83-2863-4e88-8995-356912ee98a2-credential-socket\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586947 4543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1907c83-2863-4e88-8995-356912ee98a2-istio-podinfo\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.587601 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.586959 4543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec7421a-e681-494c-989c-e780b975158f-istio-envoy\") on node \"ip-10-0-131-117.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 10:34:27.930327 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.930292 4543 generic.go:356] "Generic (PLEG): container finished" podID="d1907c83-2863-4e88-8995-356912ee98a2" containerID="8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c" exitCode=0 Nov 03 10:34:27.930533 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.930344 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" Nov 03 10:34:27.930533 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.930384 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" event={"ID":"d1907c83-2863-4e88-8995-356912ee98a2","Type":"ContainerDied","Data":"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c"} Nov 03 10:34:27.930533 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.930448 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh" event={"ID":"d1907c83-2863-4e88-8995-356912ee98a2","Type":"ContainerDied","Data":"cab1923dbb23f4acfef1ed1e5bf8a0291b5438818d69a8070bb6526e6d7cbdcb"} Nov 03 10:34:27.930533 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.930478 4543 scope.go:122] "RemoveContainer" containerID="8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c" Nov 03 10:34:27.931763 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.931742 4543 generic.go:356] "Generic (PLEG): container finished" podID="a37aaef4-8a1f-4f2d-a97e-558328072820" containerID="5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d" exitCode=0 Nov 03 10:34:27.931903 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.931773 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" event={"ID":"a37aaef4-8a1f-4f2d-a97e-558328072820","Type":"ContainerDied","Data":"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d"} Nov 03 10:34:27.931903 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.931796 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" Nov 03 10:34:27.931903 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.931830 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl" event={"ID":"a37aaef4-8a1f-4f2d-a97e-558328072820","Type":"ContainerDied","Data":"a11bd6c1823e79f215d9e4386dfa2bbcc70b6e9aa21359578fca28a9bb98bd88"} Nov 03 10:34:27.933339 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.933316 4543 generic.go:356] "Generic (PLEG): container finished" podID="bec7421a-e681-494c-989c-e780b975158f" containerID="8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243" exitCode=0 Nov 03 10:34:27.933468 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.933367 4543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" Nov 03 10:34:27.933468 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.933389 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" event={"ID":"bec7421a-e681-494c-989c-e780b975158f","Type":"ContainerDied","Data":"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243"} Nov 03 10:34:27.933468 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.933440 4543 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz" event={"ID":"bec7421a-e681-494c-989c-e780b975158f","Type":"ContainerDied","Data":"ee1b0908435bf6c5fa93c0074cfde3e77865cb467011964f3ecf382551c2821a"} Nov 03 10:34:27.940250 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.940232 4543 scope.go:122] "RemoveContainer" containerID="8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c" Nov 03 10:34:27.940565 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:27.940538 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c\": container with ID starting with 8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c not found: ID does not exist" containerID="8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c" Nov 03 10:34:27.940692 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.940571 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c"} err="failed to get container status \"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c\": rpc error: code = NotFound desc = could not find container \"8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c\": container with ID starting with 8cc94983b41a613e9a8f6163f70eaea8b05f7d1373de2bb28e6c7a12fc83bf0c not found: ID does not exist" Nov 03 10:34:27.940692 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.940590 4543 scope.go:122] "RemoveContainer" containerID="5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d" Nov 03 10:34:27.949940 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.949921 4543 scope.go:122] "RemoveContainer" containerID="5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d" Nov 03 10:34:27.950186 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:27.950166 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d\": container with ID starting with 5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d not found: ID does not exist" containerID="5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d" Nov 03 10:34:27.950262 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.950196 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d"} err="failed to get container status \"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d\": rpc error: code = NotFound desc = could not find container \"5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d\": container with ID starting with 5eb55e43bb35fd91500a574529466c6a808aedcc2403ea5370839a343be2462d not found: ID does not exist" Nov 03 10:34:27.950262 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.950211 4543 scope.go:122] "RemoveContainer" containerID="8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243" Nov 03 10:34:27.959270 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.959253 4543 scope.go:122] "RemoveContainer" containerID="8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243" Nov 03 10:34:27.959535 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:27.959518 4543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243\": container with ID starting with 8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243 not found: ID does not exist" containerID="8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243" Nov 03 10:34:27.959591 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:27.959544 4543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243"} err="failed to get container status \"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243\": rpc error: code = NotFound desc = could not find container \"8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243\": container with ID starting with 8e79b71c5a7e1dea3ee21b12c1a67bb4b21e4dedd9670a4c5687a2a8eccb3243 not found: ID does not exist" Nov 03 10:34:28.099983 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.099951 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:34:28.150782 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.150749 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9tmmz"] Nov 03 10:34:28.264393 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.264304 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 10:34:28.271446 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.271420 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7s6wh"] Nov 03 10:34:28.384624 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.384586 4543 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:34:28.393931 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.393902 4543 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-h9rgl"] Nov 03 10:34:28.452466 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.452433 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37aaef4-8a1f-4f2d-a97e-558328072820" path="/var/lib/kubelet/pods/a37aaef4-8a1f-4f2d-a97e-558328072820/volumes" Nov 03 10:34:28.452992 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.452973 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec7421a-e681-494c-989c-e780b975158f" path="/var/lib/kubelet/pods/bec7421a-e681-494c-989c-e780b975158f/volumes" Nov 03 10:34:28.453436 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.453421 4543 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1907c83-2863-4e88-8995-356912ee98a2" path="/var/lib/kubelet/pods/d1907c83-2863-4e88-8995-356912ee98a2/volumes" Nov 03 10:34:28.727297 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.727269 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:34:28.945829 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:28.945803 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:34:29.381414 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:29.381383 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:34:29.821626 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:29.821554 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:34:35.559914 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:34:35.559875 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:34:35.988083 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:35.988033 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:34:37.237219 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:37.237179 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:34:37.364477 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:37.364452 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:34:37.444181 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:37.444130 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n26bm_27e5037f-1bbd-4170-8108-6a06e3f782f2/ovn-acl-logging/0.log" Nov 03 10:34:46.545902 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:46.545864 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:34:47.212986 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:47.212950 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:34:48.641512 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:48.641479 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:34:48.858708 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:48.858681 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:34:49.623897 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:49.623853 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:34:57.040030 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:34:57.039996 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:35:05.560516 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:35:05.560477 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:35:05.978044 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:05.978011 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:35:06.195373 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:06.195342 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:35:06.634317 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:06.634290 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:35:07.068878 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:07.068800 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:35:35.563920 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:35:35.563857 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor" Nov 03 10:35:35.654824 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:35.654790 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-69hhr_98880443-bd5a-416c-8e86-2eb9a2d8fb40/manager/0.log" Nov 03 10:35:35.870864 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:35.870841 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-5w7h4_a4205e96-340e-40e7-8071-24d8206086ea/kuadrant-console-plugin/0.log" Nov 03 10:35:36.328830 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:36.328749 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:35:36.583812 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:36.583730 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-nrqft_209e8786-a2b0-454f-89b8-2b491daec379/manager/0.log" Nov 03 10:35:47.561661 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:47.561574 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:35:53.186895 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:53.186860 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-q8fhg_552cd4fb-443b-418b-a382-f9af70e6635f/main/0.log" Nov 03 10:35:53.403391 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:53.403334 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-2nzfm_b664a803-96f3-43fc-bf22-fb740e6bbef7/main/0.log" Nov 03 10:35:53.837589 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:53.837559 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xtttj_b22e17fb-2005-42e7-ab1e-31c31d3d7606/main/0.log" Nov 03 10:35:54.270558 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:35:54.270531 4543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tq9b2_d06e15c0-3989-40f0-b3b8-dfb108624b4e/main/0.log" Nov 03 10:36:01.739227 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:36:01.739189 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:36:01.861801 ip-10-0-131-117 kubenswrapper[4543]: I1103 10:36:01.861766 4543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9rkp2_e0527392-1028-47e0-a542-80038e79f6da/konnectivity-agent/0.log" limit=1024 Nov 03 10:36:05.560735 ip-10-0-131-117 kubenswrapper[4543]: E1103 10:36:05.560694 4543 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pd45j" containerName="extractor"